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Results  in  Brief 

Naval  Air  Systems  Command  Needs  to  Improve 
Management  of  Waiver  Requests 


May  15,  2015 


Findings  (cant'd) 


Objective 

Our  objective  was  to  evaluate  the 
Department  of  the  Navy's  overall 
management  of  waivers  and  deferrals 
from  operational  test  requirements  for 
Navy  Systems.  We  evaluated  the  processes 
to  justify,  review,  and  approve  requests 
for  waiver  of  criteria  to  certify  readiness 
for  operational  testing  and  deferral  of 
operational  testing  requirements  at  the 
Naval  Air  Systems  Command  (NAVAIR). 

This  report  is  the  first  in  a  series  of  reports 
that  will  evaluate  the  Navy's  management  of 
waivers  and  deferrals  for  weapons  systems. 

Findings 

NAVAIR  program  managers  did  not  fully 
implement  Navy  policy  to  request  waivers 
and  to  certify  program  readiness  for  initial 
operational  test  and  evaluation  (lOT&E) 
on  the  P-8A  Poseidon  aircraft.  Distributed 
Targeting  System,  and  E-2D  Advanced 
Hawkeye  aircraft  programs.  Specifically, 
program  managers  did  not: 

•  request  waivers  for  two  programs  that 
had  not  met  all  lOT&E  certification 
criteria,  and 

•  clearly  document  what  was  done  to 
meet  each  certification  criteria  for  a 
third  program. 

These  conditions  occurred  because  Navy 
policy  did  not  clearly  and  concisely  state 
program  managers  must  request  a  waiver 
whenever  a  program  did  not  meet  all 


criteria  required  to  begin  lOT&E.  Because  program  managers 
did  not  request  waivers,  the  program  executive  officer  could 
not  effectively  consider  program  readiness  for  entering  the 
lOT&E  test  phase.  As  a  result,  two  NAVAIR  programs  entered 
and  completed  lOT&E  with  unresolved  deficiencies  affecting 
threat  detection,  maneuverability,  information  exchange,  and 
reliability.  These  deficiencies  diminished  the  systems'  abilities 
to  perform  their  missions. 

Weapon  system  sponsors  within  the  Office  of  the  Chief  of 
Naval  Operations  (CNO)  did  not  designate  characteristics 
most  critical  to  providing  an  effective  military  capability  as 
primary  requirements  when  writing  aircraft  requirements 
documents.  Additionally,  the  Joint  Requirements  Oversight 
Council  (JROC)  review  and  validation  of  those  aircraft 
requirements  documents  did  not  result  in  focusing  the 
primary  requirements  on  the  accomplishment  of  critical 
aircraft  missions.  These  conditions  occurred  because  the 
Joint  Chiefs  of  Staff  (JCS)  guidance  did  not  specifically  require 
the  sponsors  and  JROC  to  make  sure  systems  requirements 
documents  capture  characteristics  most  critical  to  meeting 
mission  requirements  as  primary  requirements.  As  a  result, 
weapon  system  sponsors  concurred  with  the  request  for 
waivers  without  giving  the  JROC  the  opportunity  to  assess 
the  effect  of  those  waivers  on  the  military  usefulness  of  the 
systems.  For  example,  on  the  P-8A  Poseidon  aircraft  the 
sponsor  granted  a  flight  deficiency  waiver  that  diminished 
mission  capabilities  for  Anti-Submarine  and  Anti-Surface 
Warfare  and  Intelligence  gathering.  Additionally,  because 
the  requirements  documents  did  not  capture  the  critical 
system  characteristics  as  primary  requirements,  program 
managers  accepted  production  units  with  diminished  mission 
effectiveness.  For  example,  at  the  time  of  the  FRP  decision, 
the  program  manager  for  the  P-8A  Poseidon  had  accepted 
13  aircraft  for  $2.6  billion  that  could  not  fully  perform 
primary  missions. 
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Results  in  Brief 

Naval  Air  Systems  Command  Needs  to  Improve 
Management  of  Waiver  Requests 


Recommendations 

Among  others,  we  recommend  the  Assistant  Secretary 
of  the  Navy  for  Research,  Development,  and  Acquisition 
and  the  Chief  of  Naval  Operations  update  Navy  policy 
to  require  program  managers  to  request  waivers  if  they 
do  not  meet  all  criteria  required  to  certify  readiness 
for  lOT&E;  and  require  weapon  systems  sponsors  for 
acquisition  programs  of  interest  to  the  Joint  Capability 
Board  or  the  JROC,  to  certify  to  the  JCS  that  approved 
waivers  do  not  adversely  impact  primary  system 
requirements  or,  if  there  is  adverse  impact  that  the 
system  continues  to  have  military  value. 

We  recommend  that  JCS  modify  its  guidance 
on  writing  and  reviewing  system  requirements 
documents  to  require  sponsors  to  include  system 
characteristics  most  critical  to  mission  effectiveness 
as  primary  requirements. 


Management  Comments 
and  Our  Response 

The  Deputy  Department  of  the  Navy  Test  and 
Evaluation  Executive  responding  for  the  Assistant 
Secretary  of  the  Navy  for  Research,  Development,  and 
Acquisition  and  the  Chief  of  Naval  Operations,  agreed 
with  Recommendation  A.l.  The  Deputy,  responding 
for  the  Assistant  Secretary  of  the  Navy  for  Research, 
Development,  and  Acquisition,  disagreed  with 
Recommendation  B.2,  and  his  comments  did  not  fully 
address  the  specifics  of  the  recommendation.  Therefore, 
we  request  that  the  Deputy  reconsider  his  response  to 
recommendation  B.2  and  provide  additional  comments. 
The  Vice  Chairman,  Joint  Chiefs  of  Staff  agreed  with 
Recommendation  B.l  and  no  further  comments  are 
required.  Please  see  the  Recommendations  Table  on  the 
next  page 
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Recommendations  Table 


Management 

Recommendations 
Requiring  Comment 

No  Additional 
Comments  Required 

Vice  Chairman,  Joint  Chiefs  of  Staff 

B.l 

Assistant  Secretary  of  the  Navy  for  Research,  Development, 
and  Acquisition 

B.2 

A.l 

Chief  of  Naval  Operations,  Director,  Innovation,  Test  and 
Evaluation,  and  Technology, 

A.l 

Please  provide  Management  Comments  by  June  15,  2015. 
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INSPECTOR  GENERAL 

DEPARTMENT  OF  DEFENSE 

4800  MARK  CENTER  DRIVE 
ALEXANDRIA,  VIRGINIA 22350-1 500 


May  15,  2015 


MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  ACQUISITION, 

TECHNOLOGY,  AND  LOGISTICS 
VICE  CHAIRMAN  OF  THE  JOINT  CHIEFS  OF  STAFF 
DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 
NAVAL  INSPECTOR  GENERAL 

SUBJECT:  Naval  Air  Systems  Command  Needs  to  Improve  Management  of  Waiver  Requests 
(Report  No.  DODIG-2015-122] 

We  are  providing  this  report  for  review  and  comment.  We  determined  that  the  Naval  Air 
Systems  Command  needs  to  improve  management  of  waiver  requests  to  allow  the  program 
executive  officer  to  better  determine  program  readiness  for  entering  testing  to  support 
the  full-rate  production  decision.  Also,  weapon  system  sponsors  did  not  fully  define  the 
aircraft  system  characteristics  most  critical  for  providing  an  effective  military  capability 
and  designate  them  as  primary  requirements  when  writing  system  requirements  documents. 
Because  the  requirements  documents  did  not  capture  the  most  critical  system  characteristics 
as  primary  requirements,  program  managers  accepted  production  units  with  diminished 
mission  effectiveness.  For  example,  at  the  time  of  the  full-rate  production  decision,  the 
program  manager  for  the  P-8A  Poseidon  had  accepted  13  aircraft  for  $2.6  billion  that 
could  not  fully  perform  primary  missions  for  Anti-Submarine  and  Anti-Surface  Warfare 
and  Intelligence  gathering.  We  conducted  this  audit  in  accordance  with  generally  accepted 
government  auditing  standards. 

We  considered  management  comments  on  a  draft  of  this  report  when  preparing  the  final 
report.  DoD  Instruction  7650.03  requires  that  recommendations  be  resolved  promptly. 
Comments  from  the  Vice  Chairman,  Joint  Chiefs  of  Staff  addressed  the  specifics  of  the 
recommendation.  Comments  from  the  Deputy  Department  of  the  Navy  Test  and  Evaluation 
Executive,  responding  for  the  Assistant  Secretary  of  the  Navy  for  Research,  Development, 
and  Acquisition  and  the  Chief  of  Naval  Operations  agreed  with  Recommendation  A.l  but 
disagreed  with  Recommendation  B.2.  Therefore,  we  request  additional  comments  on 
Recommendation  B.2  by  June  15,  2015. 

Please  provide  comments  that  state  whether  you  agree  or  disagree  with  the  findings  and 
recommendations.  If  you  agree  with  our  recommendations,  describe  what  actions  you  have 
taken  or  plan  to  take  to  accomplish  the  recommendations  and  include  the  completion  dates 
of  your  actions.  If  you  disagree  with  the  recommendations  or  any  part  of  them,  please  give 
specific  reasons  why  you  disagree  and  propose  alternative  action  if  that  is  appropriate. 

You  should  also  comment  on  the  internal  control  weaknesses  discussed  in  the  report. 
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Please  send  a  PDF  file  containing  your  comments  to  audapi@dodig.mil.  Copies  of  your 
comments  must  have  the  actual  signature  of  the  authorizing  official  for  your  organization. 

We  cannot  accept  the  /Signed/  symbol  in  place  of  the  actual  signature.  If  you  arrange  to  send 
classified  comments  electronically,  you  must  send  them  over  the  SECRET  Internet  Protocol 
Router  Network  [SIPRNET). 

Please  direct  questions  to  me  at  (703)  604-9077  (DSN  664-9077). 


^Jac(^eline  L.  Wicecarver 
^  Assistant  Inspector  General 
Acquisition,  Parts,  and  Inventory 
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Introduction 


Introduction 


Objective 

Our  objective  was  to  evaluate  the  Department  of  the  Navy’s  overall  management 
of  waivers  and  deferrals  from  operational  test  requirements  for  Navy  systems.  We 
evaluated  the  processes  to  justify  review,  and  approve  requests  for  waiver  of  criteria  to 
certify  readiness  for  operational  testing  and  deferral  of  operational  testing  requirements 
within  the  Naval  Air  Systems  Command  (NAVAIR).  We  did  not  find  problems  with 
NAVAIR’s  management  of  deferral  requests  for  the  programs^  we  reviewed.  This 
report  is  the  first  in  a  series  of  reports  that  will  evaluate  Navy  management  of  waivers 
and  deferrals  for  weapon  systems.  See  the  Appendix  for  a  discussion  of  our  scope 
and  methodology  and  prior  coverage  related  to  the  audit  objectives. 

Navy  Policy  on  Waivers  and  Deferrals 

Secretary  of  the  Navy  Instruction  5000. 2 E 

Secretary  of  the  Navy  Instruction  [SECNAVINST)  5000.2E^  establishes  criteria  to  certify 
Navy  systems  as  ready  to  enter  into  initial  operational  test  and  evaluation  (lOT&E).^ 
lOT&E  precedes  the  full-rate  production  (final  production]  decision.  The  instruction 
requires  the  Systems  Command  commander,  program  executive  officer  (PEG),  and 
program  manager  to  conduct  an  operational  test  readiness  review  [OTRR]  to 
certify  system  readiness  to  begin  lOT&E. 

An  OTRR  is  a  product  and  process  assessment  to  make  sure  that  a  system  can 
proceed  into  lOT&E  with  a  high  probability  of  successfully  completing  operational 
testing.  Upon  completing  the  OTRR,  if  the  System  Command  commander,  PEO,  and 
program  manager  determine  the  system  is  ready  to  enter  lOT&E,  they  must  either 
certify  to  the: 

•  Commander,  Operational  Test  and  Evaluation  Force  that  the  system  is  ready 
for  lOT&E,  as  required  by  the  Test  and  Evaluation  Master  Plan  [TEMP],  with 
no  waivers  or  deferrals  requested,  or 

•  Chief  of  Naval  Operations  [CNO]  (N84],  Director  Innovation,  Test  and 
Evaluation,  and  Technology  Requirements  that  the  system  is  ready  for 
lOT&E,  with  requests  for  waivers  or  deferrals. 


^  of  the  three  programs  we  reviewed,  only  the  program  manager  for  the  P-8A  Poseidon  had  requested  deferrals  of 
operational  test  requirements. 

^  Secretary  of  the  Navy  Instruction  (SECNAVINST)  5000. 2E,  "Department  of  the  Navy  Implementation  and  Operation  of 
the  Defense  Acquisition  System  and  the  Joint  Capabilities  Integration  and  Development  System,"  September  1,  2011. 

^  Initial  Operational  Test  and  Evaluation  is  the  dedicated  operational  test  and  evaluation  conducted  on  production 
representative  articles,  to  determine  whether  systems  are  operationally  effective,  and  suitable  to  support  a  final 
production  decision. 
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SECNAVINST  5000. 2E  also  states  that,  when  waiver  or  deferral  requests  are 
anticipated,  the  program  manager  must  coordinate  with  the  program  sponsor; 

CNO  (N84],  Director,  Innovation,  Test,  and  Evaluation;  and  the  Commander, 
Operational  Test  and  Evaluation  Force  before  the  OTRR.  Additionally,  when  the 
System  Command  commander,  PEO,  and  program  manager  certify  system  readiness 
for  lOT&E  with  waivers  or  deferrals,  they  must  provide  the  program  sponsor 
within  the  Office  of  the  CNO  an  information  copy  of  the  certification.  The  program 
sponsor  must  then  formally  concur  with  the  proposed  waivers  or  deferrals. 
Concurrence  with  waivers  and  deferrals  effects  program  execution  as  follows: 

•  Waivers  are  a  deviation  from  the  criteria  identified  for  certifying  lOT&E 
readiness.  Waivers  allow  programs  to  start  lOT&E  without  meeting 
one  or  more  of  the  20  criteria  SECNAVINST  5000. 2E  requires  to  certify 
readiness  to  enter  lOT&E.  Waivers  do  not  change  or  delay  any  system  or 
testing  requirements. 

•  Deferrals  allow  programs  to  delay  the  testing  of  requirements  identified 
in  the  TEMP,  moving  testing  requirements  from  lOT&E  to  a  later  follow-on 
test  period. 

Waiver  and  deferral  approvals  can  result  in  more  rapid  delivery  of  capabilities 
to  operating  Navy  Forces.  However,  the  System  Command  commander,  PEO,  and 
program  manager  must  fully  evaluate  the  potential  impacts  waivers  and  deferrals 
have  on  the  mission  capability  delivered.  Waiver  criteria  to  certify  readiness 
for  lOT&E  or  defer  operational  test  requirements  could  result  in  premature  final 
production  decisions.  This  could  create  the  need  for  the  costly  retrofit  of  fielded 
units  to  deliver  operational  performance  that  is  less  than  required  to  fully  meet 
the  expected  threat.  CNO  N84  staff  stated  that  their  "approval"  of  waivers  and 
deferrals  indicates  that  the  program  manager  followed  the  OTRR  process  necessary 
to  certify  program  readiness  for  lOT&E  despite  risks. 

Secretary  of  the  Navy  Manual  M-5000.2 

Secretary  of  the  Navy  Manual,  M-5000.2,^  provides  additional  discretionary 
guidance  relating  to  the  Navy's  management  of  waivers  and  deferrals. 


^  Secretary  of  the  Navy  Manual  M-5000.2,  "Acquisition  and  Capabilities  Guidebook,"  May  9,  2012,  Section  4.6, 
"Certification  of  Readiness  for  Operational  Testing." 
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Naval  Air  Systems  Command 

NAVAIR  provides  support  (people,  processes,  tools,  training,  mission  facilities,  and 
core  technologies]  to  Naval  Aviation  PEOs  and  their  assigned  program  managers  to 
help  them  meet  the  cost,  schedule,  and  performance  requirements  of  their  assigned 
acquisition  programs.  NAVAIR’s  support  includes: 

•  research; 

•  system  design  and  development; 

•  acquisition; 

•  test  and  evaluation; 

•  training  facilities  and  equipment; 

•  repair  and  modifications;  and 

•  in-service  engineering  and  logistics  support. 

Background  on  Programs  Selected  for  Review 

We  identified  three  NAVAIR  acquisition  programs  that  received  final  production 
decisions  from  April  14,  2012,  through  April  14,  2014,  the  P-8A  Poseidon  aircraft, 
E-2D  Advanced  Hawkeye  aircraft,  and  Distributed  Targeting  System  (DTS).  The 
following  sections  describe  those  programs. 

P-8A  Poseidon  Aircraft 

The  P-8A  Poseidon  aircraft  program  is  an  Acquisition  Category  ID^  major  defense 
acquisition  program  that  had  its  final  production  decision  on  January  3,  2014.  The 
Navy  designed  the  P-8A  Poseidon  to  replace  the  aging  P-3C  Orion  aircraft.  Like  the 
P-3C;  the  P-8A  aircraft  provides  capabilities  for  three  principal  missions: 

•  anti-submarine  warfare; 

•  anti-surface  warfare;  and 

•  intelligence,  surveillance,  and  reconnaissance  in  maritime  operations. 

The  Capability  Production  Document^  [the  system  requirements  document)  states 
that  the  anti-submarine  warfare  mission,  which  detects,  tracks,  and  destroys  or 
neutralizes  hostile  submarines,  was  the  primary  reason  the  Navy  invested  in  the 
P-8A  aircraft.  The  anti-surface  warfare  mission  provides  maritime  superiority 


^  Acquisition  category  ID  is  a  program  for  which  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
estimates  eventual  total  expenditure  for  research,  development,  test,  and  evaluation  of  more  than  $480  million  in 
FY  2014  constant  dollars  or,  for  procurement,  of  more  than  $2.79  billion  in  FY  2014  constant  dollars. 

^  "Capability  Production  Document  for  the  United  States  Navy  P-8A  Poseidon  Multi-Mission  Maritime  Aircraft  (MMA)," 
Increment  1,  June  22,  2009. 
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against  surface  vessels  and  a  common  sea  surface  picture.  The  intelligence, 
surveillance,  and  reconnaissance  mission  provides  a  flexible  and  responsive 
intelligence  gathering  capability  in  support  of  Joint,  Naval,  and  National  interests. 

The  Navy  sought  to  expedite  procuring  and  deploying  the  P-8A  Poseidon  because  of 
concerns  with  airframe  corrosion  on  the  P-3C  Orion.  While  P-3C  Orion  fatigue  has 
remained  a  persistent  risk,  the  Navy  has  inspection,  repair,  and  modification  efforts 
in  place  to  sustain  the  P-3C  Orion  fleet  until  the  P-8A  Poseidon  began  replacing  the 
P-3C  Orion  in  2013.  As  discussed  in  DODIG  Report  2013-088,^  the  P-8A  Poseidon 
had  uncorrected  system  deficiencies  that  impacted  the  above  mission  capabilities 
entering  lOT&E.  However,  Navy  managers  accepted  the  risk  from  diminished 
capabilities  and  allowed  the  program  to  enter  into  lOT&E,  without  having  to  fully 
correct  the  deficiencies  until  after  the  full-rate  production  decision. 

E-2D  Advanced  Hawkeye  Aircraft 

The  E-2D  Advanced  Hawkeye  aircraft  program  is  an  Acquisition  Category  IC 
major  defense  acquisition  program  that  had  its  final  production  decision  on 
March  1,  2013.  The  E-2D  aircraft  is  an  all-weather,  twin-engine,  carrier-based, 
airborne  command,  control,  and  surveillance  aircraft  designed  to  extend  task  force 
defense  boundaries.  The  aircraft's  mission  includes: 

•  advance  warning  of  approaching  enemy  surface  units  and  aircraft; 

•  guide  interceptor  or  attack  aircraft; 

•  area  surveillance,  intercept,  and  search  and  rescue; 

•  communications  relay;  and 

•  air  traffic  control. 

The  E-2D  aircraft  replaces  the  E-2C,  multi-mission  airborne  early  warning  and 
airborne  battle  management  command  and  control  aircraft. 

Distributed  Targeting  System 

The  DTS  is  an  Acquisition  Category  IIP  program  that  had  its  final  production 
decision  on  April  19,  2013.  The  DTS  is  designed  to  improve  the  attack  accuracy 
using  on-board  processing  of  the  F/A-18E/F  Super  Hornet  aircraft.  The  DTS 
compares  the  geographic  imagery  of  the  ground  below  from  aircraft  sensors 
against  reference  imagery  in  a  database  at  National  Geospatial-Intelligence  Agency. 
DTS  generates  a  match  based  on  similarities  between  the  imageries  to  generate 


(FOUO)  Report  No.  DODIG-2013-088,  "The  Navy  P-8A  Poseidon  Aircraft  Needs  Additional  Critical  Testing  Before  the 
Full-Rate  Production  Decision/' June  10,  2013. 

^  Acquisition  Category  III  is  an  acquisition  program  for  which  the  DoD  Component  Head  estimates  eventual  total 

expenditure  for  research,  development,  test,  and  evaluation  of  less  than  $185  million  in  FY  2014  constant  dollars  or,  for 
procurement,  of  less  than  $835  million  in  FY  2014  constant  dollars. 
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targeting  coordinates.  This  capability  will  better  enable  the  F/A-18E/F  aircraft  to 
perform  air  controller,  attack  coordination,  and  reconnaissance  missions  and  to 
conduct  attacks  on  stationary  land  targets,  such  as  mobile  missile  launchers. 

Review  of  Internal  Controls 

DoD  Instruction  5010.40,  "Managers’  Internal  Control  Program  Procedures," 

May  30,  2013,  requires  DoD  organizations  to  implement  a  comprehensive  system  of 
internal  controls  that  provides  reasonable  assurance  that  programs  are  operating 
as  intended  and  to  evaluate  the  effectiveness  of  the  controls.  We  identified  internal 
control  weaknesses  in  NAVAIR  implementation  of  SECNAVINST  5000. 2E  policy  for 
requesting  waivers  and  certifying  program  readiness  for  lOT&E.  We  also  identified 
internal  control  weaknesses  in  CNO  and  Joint  Chiefs  of  Staff  (JCS)  procedures  to 
identify,  as  primary  system  requirements,  aircraft  system  characteristics  most 
critical  for  effective  military  capability  in  system  requirements  documents.  These 
control  weaknesses  allowed  program  sponsors  to  concur  with  the  requests  for 
waivers  without  a  requirement  to  discuss  with  the  JCS  the  effects  those  waivers 
would  have  on  military  utility.®  We  will  provide  a  copy  of  the  report  to  the  senior 
official  responsible  for  internal  controls  in  the  Navy. 


®  A  system  designed  for  or  possessing  a  number  of  useful  or  practical  purposes  rather  than  a  single,  specialized  one. 
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Finding  A 

Naval  Air  Systems  Command  Needs  to  Improve 
Management  of  Waiver  Requests 

NAVAIR  program  managers  did  not  fully  implement  Navy  policy  to  request 
waivers  and  certify  program  readiness  for  lOT&E  on  the  P-8A  aircraft,  DTS, 
and  E-2D  aircraft  programs.  Specifically,  program  managers  did  not: 

•  request  waivers  for  the  P-8A  and  DTS  when  those  programs  did  not  meet 
all  lOT&E  certification  criteria;  and 

•  clearly  document  what  was  done  to  meet  each  of  the  certification  criteria 
for  the  E-2D. 

These  conditions  occurred  because  Navy  policy  did  not  clearly  and  concisely  state 
that  program  managers  must  request  a  waiver  whenever  the  program  did  not  meet  all 
criteria  required  to  enter  lOT&E.  Because  program  managers  did  not  request  waivers, 
the  PEO  could  not  effectively  consider  program  readiness  for  entering  the  lOT&E  test 
phase.  As  a  result,  the  P-8A  and  the  DTS  program  entered  and  completed  lOT&E 
with  unresolved  deficiencies  affecting  threat  detection,  maneuverability,  information 
exchange,  and  reliability.  These  deficiencies  diminished  the  ability  of  the  P-8A  aircraft 
and  the  DTS  to  perform  their  primary  missions. 


Program  Managers  Need  to  Request  Waivers  for  All 
Certification  Requirements  Not  Met 

The  P-8A  program  manager  did  not  meet  4  of  the  20  criteria  specified  in 
SECNAVINST  5000. 2E  to  certify  readiness  for  lOT&E.  Although  the  P-8A  program 
manager  requested  a  waiver  from  the  flight  deficiency  criteria,  a  waiver  was  not 
requested  for  three  additional  criteria.  Additionally,  the  DTS  program  manager  did 
not  request  a  waiver  for  one  of  the  certification  criteria  that  the  program  had  not 
met.  Specifically,  the  program  managers  did  not  request  waivers  when: 

•  system  performance  requirements  in  the  test  and  evaluation  master 
plan  (TEMP)  were  not,  or  were  not  projected  to  be,  satisfied; 

•  necessary  logistics  support  (spares,  repair  parts,  and  equipment)  was  not 
expected  to  be  available  for  lOT&E;  and 

•  the  Joint  Interoperability  Test  Command  (JITC)  had  not  concurred  that 
program  interoperability^  was  sufficient  to  enter  lOT&E. 


“  The  ability  to  support  military  operations  and  effectively  exchange  information. 


6  I  DODIG-2015-122 


FOR  OFFICIAL  USE  ONLY 


FOR  OFFICIAL  USE  ONLY 


Finding  A 


Meeting  Performance  Requirements  Criterion 

The  P-8A  and  the  DTS  program  managers  did  not  request  waivers  when 
their  programs  did  not  meet  performance  requirements  in  the  TEMP. 
SECNAVINST  5000. 2E  requires  programs  to  demonstrate,  through  test  and 
evaluation  performance,  that  requirements  identified  in  the  TEMP  have  been 
satisfied  or  are  projected  to  meet  system  requirements. 


P-8A  Poseidon  Primary  Performance  Requirements  Not  Met 
The  program  manager  for  the  P-8A  did  not  request  a  waiver  from  meeting  test 
and  evaluation  results  criteria  even  though  the  Operational  Test  Readiness 
Review  (OTRR]  checklist  summary  showed  that  the  criteria  were  not  met.  The 
briefing  charts  identified  that  12  critical  technical  parameters  and  8  measures 
of  effectiveness  were  not  met  at  the  OTRR  on  August  28,  2012.  The  Transition 
Report”  was  issued  more  than  a  year  later  to  support  follow-on  test  and  evaluation 
of  the  P-8A.  The  report  showed  that  the  P-8A  did  not  meet  performance  thresholds 
for  four  critical  technical  parameters  and  two  measures  of  effectiveness. 

•  Critical  technical  parameters  are  a  measureable  critical  system 
characteristic  that,  when  achieved,  demonstrate  that  the  program  has 
achieved  a  desired  operational  performance  capability. 

•  Measures  of  effectiveness  are  the  data  used  to  measure  the  mission 
accomplishment  that  comes  from  the  use  of  the  system  data  in  its 
expected  environment. 


For  example,  the  P-8A  could  not  detect”  frigates  (small,  fast 
military  ships]  at  110  nautical  miles,  which  was  one  of  the  ^ 
critical  technical  parameters.  Also,  the  P-8A  aircraft 
that  was  assigned  to  the  fleet  was  not  cleared  to 
perform  aircraft  maneuvers  at  the  limited  load  (force 
of  gravity]  capacity,  which  the  system  requirements 
document  mandated  for  the  anti-submarine  warfare 
mission.  However,  program  office  staff  stated  that 
the  P-8A  could  satisfactorily  perform  the  anti-submarine 
mission  while  operating  at  a  lessor  force  of  gravity  capacity. 


P-8A  could 
not  detect 
frigates  (small, 
fast  military  ships] 
at  110  nautical  miles, 
which  was  one  of  the 
critical  technical 
parameters. 


"P-8A  Poseidon  Aircraft  Integrated  Test  (IT-5)  Technical  Evaluation  in  Support  of  Operational  Test  Readiness 
Review  (OTRR)  and  Transition  to  Follow-On  Operational  Test  and  Evaluation  (OT-C2)/'  October  2,  2013. 

To  classify  and  identify  hostile  surface  craft  in  support  of  the  anti-surface  warfare  mission. 


FOR  OFFICIAL  USE  ONLY 


DODIG-2015-122  |  7 


Finding  A 


FOR  OFFICIAL  USE  ONLY 


Distributed  Targeting  System  Performance  Requirement  Not  Met 
(FOUO)  The  DTS  program  manager  did  not  request  a  waiver  from  the  system 
performance  for 

Acquisition  Decision  Memorandum  stated  that  the  DTS  program  manager  would 
not  award  a  contract  for  a  second  low-rate  initial  production  lot  until  the  average 
flight  hours  between  operational  mission  failures  had  improved  and  was  no  longer 
a  high  risk. 

Additionally,  the  DTS  Transition  ReporF^  states  the  DTS  demonstrated  performance 
levels  for  all  tested  requirements  that  met  or  exceeded  threshold  requirements 
(minimum  level  of  performance  required)  except  for  the  average  flight  hours 
between  operational  mission  failures.  The  DTS  Transition  Report  also  stated  the 
system  demonstrated  an  average  of  111  flight  hours,  instead  of  the  minimum 
requirement  of  130  flight  hours  for  average  flight  hours  between  operational 
mission  failures.  According  to  program  office  staff,  at  the  time  of  the  OTRR, 
the  DTS  demonstrated  an  average  of  116.9  flight  hours  between  operational 
mission  failures  with  all  reliability  growth  curves  on  track  to  reach  the  minimum 
requirement  of  130  flight  hours  before  initial  operational  capability.  In  addition, 
the  program  office  staff  stated  in  their  comments  to  the  draft  report,  28  days  after 
the  OTRR,  on  July  12,  2012,  the  DTS  reached  the  required  130  flight  hours  between 
operational  mission  failures. 

However,  6  months  later,  the  Commander,  Operational  Test  and  Evaluation  Force 
reporF''^  identified  the  average  flight  hours  between  operational  mission  failures 
as  a  deficiency  that  would  have  a  "moderate  impact"  on  mission  accomplishment. 
Specifically,  the  report  stated  an  unreliable  DTS  would  result  in  a  mission  abort 
and  a  degraded  capability  to  provide  targeting  coordinates  to  warfighters  during 
amphibious  warfare  missions.  It  explained  that  overall  DTS  reliability  was  below 
the  ideal  reliability  growth  curve  for  the  system.  The  report  identified  an  average 
of  90.7  flight  hours  between  operational  mission  failures,  significantly  lower  than 
the  requirement  of  130  flight  hours  between  operational  mission  failures  specified 
in  the  DTS  TEMP. 


DT/OT  Transition  Report  for  AN/AYK-29(v)  Distributed  Targeting  System  Technical  Evaluation,  May  15,  2012. 
"Distributed  Targeting  System  (DTS)  Operational  Test  Agency  Evaluation  Report,"  January  23,  2013. 
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As  of  August  1,  2013,  almost  8  months  after  the  lOT&E  report,  the  Commander, 
Operational  Test  and  Evaluation  Force  memorandum^®  did  not  include  the  reliability 
deficiency  as  one  of  the  deficiencies  that  the  program  manager  had  corrected. 

Meeting  Necessary  Logistical  Support  Available  Criterion 

The  P-8A  program  office  documented  that  the  logistic  support  criteria  was  not  met 
in  the  OTRR  checklist  summary  but  did  not  request  a  waiver.  SECNAVINST  5000. 2E 
requires  that  logistic  support,  including  spares,  repair  parts,  and  support  and 
ground  support  equipment,  be  available  to  support  the  planned  lOT&E.  The 
P-8A  Transition  Report^®  for  lOT&E  states  system  reliability,  maintainability,  and 
availability  for  the  P-8A  system  design  and  development  was  conducted  using 
contractor  logistic  support  maintenance.  The  transition  report  for  lOT&E  also 
stated  that  all  support  products  were  evaluated  as  acceptable  except  for  spares 
and  support  equipment,  because  the  contractor  had  not  delivered  multiple  pieces  of 
support  equipment. 

Consequently,  several  logistical  support  tests  were  not  scheduled  for  completion 
until  just  before  or  shortly  after  the  start  of  lOT&E.  Although  the  program 
manager  purchased  additional  support  equipment  and  spares,  not  all  of  the  support 
equipment  and  spares  purchased  were  scheduled  for  delivery  before  the  start  of 
lOT&E.  At  the  time  of  the  OTRR,  the  program  was  at  risk  for  not  being  able  to 
complete  lOT&E  because  of  insufficient  spares  and  support  equipment. 

Meeting  Joint  Interoperability  Test  Command  Concurrence  on 
Interoperability  Progress  Criterion 

The  P-8A  aircraft  did  not  meet  the  interoperability  criterion;  however,  the 
program  manager  did  not  request  a  waiver  from  meeting  this  criterion. 
SECNAVINST  5000. 2E  requires  programs  with  interoperability  requirements  to 
have  an  approved  information  support  plan  and  JITC  concurrence  that  program 
interoperability  was  demonstrated  during  system  development  sufficiently  to  enter 
lOT&E.  While  the  P-8A  program  manager  had  an  approved  information  support 
plan/^  JITC  had  not  issued  a  certification  of  interoperability  for  the  P-8A. 


"Verification  of  Correction  of  Deficiencies  (VDC)  of  the  Distributed  Targeting  System  (DTS)/'  August  1,  2013. 

"DT/OT  Transition  Report  for  P-8A  Poseidon  Aircraft  Integrated  Test  {IT-4)  Technical  Evaluation  in  Support  of  Operational 
Test  Readiness  Review  (OTRR)  and  Transition  to  Initial  Operational  Test  and  Evaluation  (OT-Cl)/'  August  22,  2012. 

On  October  8,  2009,  the  program  manager  received  acceptance  from  the  Director,  Commercial  Technologies  and 
Systems  of  the  P-8A  Poseidon  Information  Support  Plan. 
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Program  office  staff  stated  that  JITC  did  not  express  concern  about  interoperability 
during  the  OTRR  held  on  August  28,  2012.  However,  the  JITC  Acting  Chief,  Force 
Application/Force  Protection  Portfolio,  issued  an  August  2012  memorandum^® 
which  stated, "...  the  unresolved  deficiencies  present  a  high  risk  to  mission  success, 
should  the  P-8A,  Increment  I,  proceed  to  Initial  Operational  Test 
and  Evaluation  (lOT&E)."  JITC  concerns  included  unresolved 
deficiencies  that  negatively  impacted  interoperability  in  the 
aircraft’s  execution  of  its  critical  missions.  For  example, 
limited  reception  of  satellite  communication  could  result 
in  reduced  aircrew  situational  awareness  of  emerging 
threats  during  P-8A  missions.  Additionally,  interoperability 
continued  as  a  JITC  concern  after  the  P-8A  completed  lOT&E. 

On  October  24,  2013,  a  JITC  memorandums^  granted  conditional  interoperability 
certification  to  the  P-8A.  The  memorandum  stated  that  there  were  still 
five  interoperability  certification  requirements  that  were  "partially  met," 
one  requirement  that  was  not  met,  and  one  requirement  not  tested.  The 
memorandum  explained  that  the  seven  certification  criteria  had  a  "moderate” 
operational  impact  on  the  system.  Interoperability  is  necessary  to  support  the 
aircraft’s  primary  missions. 

Operational  Test  Readiness  Review  Briefing  Charts 
Need  to  Address  Certification  Criteria 

The  OTRR  briefing  charts  from  February  1,  2012,  for  the  E-2D,  including 
back-up  charts,  did  not  show  evidence  of  how  the  program  manager  met  9  of  the 
20  certification  criteria  that  were  checked  off  on  the  OTRR  Checklist  Summary. 
However,  after  reviewing  the  "E-2D  AHE  Road  to  lOT&E  Deep  Dive"  briefing  charts 
from  January  12,  2012,  which  were  not  part  of  the  OTRR  briefing,  we  found  support 
for  those  nine  criteria.  Without  including  the  information  found  in  those  briefing 
charts,  the  OTRR  did  not  fully  inform  the  chairperson  or  other  stakeholders  as 
to  whether  the  program  should  have  entered  into  lOT&E  in  support  of  the  final 
production  decision.  However,  the  E-2D  program  office  staff  stated  that  the  PEO 
was  provided  updates  on  the  status  of  meeting  the  20  certification  criteria. 


Additionally, 
interoperability 
continued  as  a  JITC 
concern  after  the 
P-8A  completed 
lOT&E. 


18  "p_gA  Poseidon  Multi-Mission  Aircraft  (MMA),  Increment  1  Net  Ready  Key  Performance  Parameter  (NR-KPP)  Interim 
Status/'  August  2012. 

JITC  memorandum,  "Joint  Interoperability  Certification  of  the  P-8A  Multi-Mission  Maritime  Aircraft  (MMA) 
Increment  1  Build  TR-13/'  October  24,  2013. 
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The  SECNAVINST  5000. 2E  requires  acquisition  managers,  including  the  PEO 
and  the  program  manager,  to  conduct  the  OTRR  before  they  certify  program 
readiness  for  lOT&E.  The  OTRR  is  complete  when  the  PEO  evaluates  and 
determines  system  readiness  for  lOT&E.  NAVAIR  Instruction  3960.20,^°  states  the 
program  manager’s  OTRR  brief  "must  address  each  certification  criteria  listed  in 
SECNAVINST  5000. 2E.”  In  addition,  the  assistant  program  manager  for  test  and 
evaluation  with  the  leader  of  the  program’s  integrated  product  team  must  make 
sure  the  OTRR  certification  criteria  checklist  shows  which  certification  criteria  are 
met  and  not  met,  is  complete,  and  presented  to  the  chairperson  of  the  OTRR. 

Navy  Policy  and  Guidance  Needed  Clarification 

The  deficiencies  in  managing  waiver  requests  and  presenting  OTRR  briefings 
occurred,  because  policy  in  the  SECNAVINST  5000. 2E  and  Secretary  of  the 
Navy  Manual  M-5000.2  did  not  provide  adequate  clarification  and  direction  to 
implement  policy  on  waivers  and  certify  program  readiness  for  lOT&E.  Most 
significantly,  the  SECNAVINST  5000. 2E  does  not  clearly  and  concisely  state  that 
program  managers  must  send  waiver  requests  to  CNO  when  they  determine  they 
have  not  satisfied  one  or  more  of  the  certification  criteria. 

On  June  25,  2014,  staff  in  the  CNO,  Director,  Innovation,  Test  and  Evaluation, 
and  Technology  [N84)  stated  that  they  recognized  the  need  to  update 
SECNAVINST  5000. 2E  to  clarify  policy  on  managing  waiver  requests.  N84  staff 
explained  that  they  wanted  to  make  sure  acquisition  managers  send  a  message  to 
N84  to  request  waivers,  when  they  determine  that  a  system  will  not  meet  one  or 
more  of  the  20  certification  criteria  for  entering  lOT&E. 

As  a  way  forward,  N84  staff  stated  they  were  updating  SECNAVINST  5000. 2E  to 
clarify  the  section  related  to  waivers.  Specifically,  they  will  define  terms  and 
identify  what  documentation  program  managers  should  provide  in  messages  to 
request  waivers  or  deferrals.  On  August  15,  2014,  N84  staff  gave  us  the  following 
draft  revision  to  SECNAVINST  5000. 2E  that  clarified  the  requirements  for 
requesting  waivers  and  deferrals: 

If  the  System  Command  commander.  Program  Executive  Officer, 
direct  reporting  program  managers  or  program  managers  determine 
that  any  of  the  20  criteria  identified  in  paragraph  4. 3. 2. 3.1  have  not 
been  adequately  satisfied  (leading  to  a  deviation  from  SECNAV  policy 
-  formerly  known  as  a  waiver),  and/or  some  capabilities  identified 
in  the  program  requirements  documents  have  not  been  achieved 
and  are  not  available  for  testing  (deferral  of  test  requirements) 
he  or  she  will  send  a  message  to  N84  [Director,  Innovation,  Test  & 

Evaluation,  and  Technology  Requirements]  certifying  the  system 

NAVAIR  Instruction  3960. 2D,  "Acquisition  Test  and  Evaluation,"  May  30,  2012. 
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is  ready  to  proceed  to  operational  test  with  test  and  evaluation 
exceptions,  and  include  details  of  the  requested  deviation(s)  from 
SECNAV  policy(s)  and/or  deferrals  of  test  requirements.  Provide 
information  copies  to  the  program  sponsor  (Navy  only,  who  must 
provide  formal  concurrence  with  deferrals),  [Assistant  Secretary  of 
the  Navy  for  Research,  Development  and  Acquisition]  ASN(RD&A), 
operational  test  agency,  and  when  a  program  is  on  the  OSD  [Office 
of  the  Secretary  of  Defense]  Test  and  Evaluation  oversight  list,  to 
DOT&E  [Director,  Operational  Test  and  Evaluation]  (who  must  also 
formally  concur  with  deferrals). 

N84  staff  stated  that  their  proposed  policy  updates  were  in  response  to  our  audit 
of  the  P-8A  Poseidon  Aircraft  program^^  and  our  discussions  on  this  audit. 

We  reviewed  the  proposed  revision  to  the  SECNAVINST  5000. 2E  policy;  however,  it 
did  not  require  program  managers  to  show  evidence  of  what  they  have  accomplished 
to  meet  each  of  the  20  certification  criteria  to  certify  readiness  for  lOT&E  in 
the  OTRR  brief.  We  are  recommending  that  the  ASN(RDA)  and  CNO  N84  update 
SECNAVINST  5000. 2E  and  Secretary  of  the  Navy  Manual,  M-5000.2,  Section  4.6, 
“Certification  of  Readiness  for  Operational  Testing,"  to: 

•  Emphasize  that  program  managers  must  request  waivers  whenever  they 
do  not  meet  any  of  the  20  criteria  required  for  certifying  readiness  for 
initial  operational  test  and  evaluation;  and 

•  Clarify  that  OTRR  briefings  must  provide  specific  explanations  of  program 
accomplishments  against  each  of  the  20  certification  criteria  to  clearly 
document  either  that  the  criteria  has  been  met  or  that  a  waiver  or  deferral 
request  is  necessary. 

Program  Executive  Officer  Lacked  Important 
Information  for  Making  Informed  Decisions 

Because  program  managers  did  not  request  waivers,  the  PEO  could  not  effectively 
consider  program  readiness  for  entering  the  lOT&E  test  phase.  As  a  result, 
the  P-8A  and  the  DTS  program  entered  and  completed  lOT&E  with  unresolved 
deficiencies  affecting  threat  detection,  maneuverability,  information  exchange,  and 
reliability.  These  deficiencies  diminished  the  systems'  ability  to  perform  their 
missions.  For  the  P-8A,  unresolved  deficiencies  diminished  mission  capability  in 
the  aircraft's  primary  missions  of  anti-submarine  warfare;  anti-surface  warfare; 
and  intelligence,  surveillance,  and  reconnaissance.  For  DTS,  the  unresolved 
deficiencies  reduced  its  reliability  to  perform  the  amphibious  warfare  portion  of 
its  mission. 


(FOUO)  Report  No.  DODIG-2013-088,  "The  Navy  P-8A  Poseidon  Aircraft  Needs  Additional  Critical  Testing  Before  the 
Full-Rate  Production  Decision/' June  10,  2013. 
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Management  Comments  on  the  Finding 
and  Our  Response 

The  Deputy  Department  of  the  Navy  Test  and  Evaluation  Executive,  responding 
for  the  Chief  of  Naval  Operations,  Director,  Innovation,  Test  and  Evaluation,  and 
Technology,  provided  the  following  comments  on  the  finding.  For  the  full  text  of 
the  Deputy’s  comments,  see  the  Management  Comments  section  of  the  report. 

Department  of  the  Navy  Comments 

The  Deputy  disagreed  with  the  finding;  however,  the  Deputy  acknowledged  that 
the  current  SECNAVINST  5000. 2E  could  be  updated  to  provide  more  clear  guidance 
and  leadership  expectations  of  the  OTRR  process.  The  Deputy  stated  that: 

•  the  finding  implies  that  the  only  information  discussed  at  the  OTRR  is 
regarding  certification  criteria  not  met.  Additionally,  this  is  the  only 
information  provided  to  the  PEO  to  support  his  or  her  decision  during 
the  OTRR.  This  is  incorrect,  the  OTRR  is  a  culmination  of  many  months 
of  effort  by  the  program  office  and  stakeholders  to  determine  the  status 
of  the  20  certification  criteria  and  how  much  detail  will  be  included  in 
the  OTRR  briefing  charts.  In  addition,  there  are  many  layers  of  expert 
reviews  making  sure  that  the  PEO  is  provided  objective  data  to  make  an 
informed  decision  at  the  OTRR. 

•  P-8A  OTRR  provided  the  PEO  with  an  extremely  detailed  briefing 
with  several  supporting  reports  to  base  the  readiness  for  test  decision 
including  draft  waivers  and  deferrals.  The  OTRR  included  all  program 
stakeholders,  who  carefully  reviewed  the  information  and  advised  the 
PEO  on  whether  certification  criteria  were  satisfactorily  met. 

•  while  the  report  is  accurate,  the  program  manager  documented  what 
was  done  to  meet  each  of  the  certification  criteria  for  the  E-2D  aircraft 
program.  The  report  acknowledges  that  the  PEO  was  presented  with 
evidence  on  how  the  program  met  each  of  the  20  certification  criteria. 

•  while  the  report  is  accurate,  the  reliability  statements  on  the  DTS  are 
misleading.  The  DTS  reached  the  130  flight  hours  between  operational 
mission  failures  requirement  on  July  12,  2012,  28  days  after  the  OTRR. 

The  DTS  was  on  a  growth  curve  for  reliability  above  the  requirements 
identified  in  the  TEMP  and  as  such,  exceeded  the  requirement  to  enter 
into  the  lOT&E  phase. 

Our  Response 

We  recognize  that  during  the  OTRR  program  staffs  provide  the  PEO  and  other 
stakeholders  with  information  beyond  how  the  programs  accomplished  the  criteria 
to  certify  readiness  for  lOT&E  and  that  there  are  multiple  levels  of  program  review. 
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However,  when  program  managers  request  waivers,  it  serves  to  highlight  to  the 
PEO  and  other  stakeholders  specific  shortfalls  from  the  criteria  that  programs  must 
normally  meet  according  to  SECNAVINST  5000. 2E  to  demonstrate  they  are  ready  to 
enter  lOT&E.  The  Navy  has  long  recognized  the  importance  to  meet  these  lOT&E 
certification  criteria  and  to  request  waivers  for  any  deviations.  Specifically,  with 
only  minor  wording  changes,  the  same  20  certification  criteria  and  provisions  to 
request  waivers  in  the  SECNAVINST  5000. 2E  were  also  provided  in  earlier  versions 
of  this  Instruction,  including  SECNAVINST  5000. 2C,  November  19,  2004. 

We  revised  the  report  to  include  the  Deputy’s  statement  that  the  DTS  reached 
130  flight  hours  between  operational  mission  failures  requirement  on  July  12,  2012. 
However,  our  report  also  states  that  6  months  later,  the  Commander,  Operational 
Test  and  Evaluation  Force  report  identified  the  average  flight  hours  between 
operational  mission  failures  as  a  deficiency  that  would  have  a  "moderate  impact" 
on  mission  accomplishment.  Specifically,  the  report  identified  an  average  of 
90.7  flight  hours  between  operational  mission  failures,  significantly  lower  than  the 
requirement  of  130  flight  hours  between  operational  mission  failures  specified  in 
the  DTS  TEMP. 

Recommendations,  Management  Comments, 
and  Our  Response 

Recommendation  A.l 

We  recommend  that  the  Assistant  Secretary  of  the  Navy  for  Research, 

Development,  and  Acquisition  and  the  Chief  of  Naval  Operations,  Director, 
Innovation,  Test  and  Evaluation,  and  Technology,  update: 

•  Secretary  of  the  Navy  Instruction  5000. 2E,  "Department  of  the  Navy 
Implementation  and  Operation  of  the  Defense  Acquisition  System  and  the 
Joint  Capabilities  Integration  and  Development  System;"  and 

•  Secretary  of  the  Navy  Manual,  M-5000.2,  "Acquisition  and  Capabilities 
Guidebook,"  May  9,  2012,  Section  4.6,  "Certification  of  Readiness  for 
Operational  Testing,"  to: 

a.  Emphasize  that  program  managers  must  request  waivers 
whenever  they  do  not  meet  any  of  the  20  criteria  the  Secretary  of 
the  Navy  Instruction  5000. 2E  requires  programs  to  meet  to  certify 
readiness  for  initial  operational  test  and  evaluation;  and 

b.  Clarify  that  Operational  Test  Readiness  Review  briefings  to 
stakeholder  groups  should  include  specific  explanations  of 
program  accomplishments  against  each  of  the  20  certification 
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criteria  to  clearly  document  either  that  the  criteria  was  met 
or  a  waiver  or  deferral  request  was  coordinated  with  the  Chief 
of  Naval  Operations,  Director,  Innovation,  Test  and  Evaluation, 
and  Technology;  the  program  sponsors;  and  the  Commander, 
Operational  Test  and  Evaluation  Force. 

Deputy  Department  of  the  Navy  Test  and  Evaluation 
Executive  (N84C)  Comments 

The  Deputy  Department  of  the  Navy  Test  and  Evaluation  Executive,  responding  for 
the  ASN[RD&A),  agreed,  stating  that  SECNAV  will  determine  the  best  method  to 
promulgate  future  Navy  acquisition  policy.  The  Deputy  further  stated  that  while 
leadership  considers  options,  he  would  issue  interim  guidance  to  address  the 
waiver  and  deferral  process.  The  guidance  will: 

•  replace  the  term  "waiver"  with  "deviation  from  SECNAV  policy,"  requiring 
the  Program  Executive  Officer  to  notify  the  ASN(RD&A)  that  a  program 
will  proceed  to  operational  test  without  achieving  one  or  more  of  the 

20  specified  certification  criteria  for  starting  this  test  phase; 

•  replace  the  term  "deferral"  with  "deferral  of  test  requirements,”  and 
will  apply  to  a  delay  in  testing  capabilities  identified  in  the  current 
requirements  document  and  agreed-to  in  the  TEMP; 

•  emphasize  that  when  requesting  a  deviation  in  the  OTRR  certification 
message,  the  PEO  identifies  which  of  the  20  certification  criteria  were  not 
met  and  states  why  the  decision  was  made  to  proceed  to  operational  test 
without  meeting  those  specific  criteria;  and 

•  require  a  summary  in  the  OTRR  briefing  charts  of  the  program  office 
assessment  of  the  20  criteria  and  make  sure  that  each  criteria  is  discussed 
in  adequate  detail  to  support  the  program  office  assessment. 

Our  Response 

The  Deputy  addressed  the  specifics  of  the  recommendation,  and  no  further 
comments  are  required. 
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Requirements  Documents  Must  Identify  System 
Characteristics  Most  Critical  to  Providing  an  Effective 
Military  Capability 

Weapon  system  sponsors  within  the  Office  of  the  CNO  did  not  fully  define 
the  aircraft  system  characteristics  most  critical  for  providing  an  effective 
military  capability.  In  addition,  they  did  not  designate  the  characteristics  as 
key  performance  parameters  (primary  requirements)  when  writing  system 
requirements  documents  for  the  P-8A  and  E-2D  aircraft  programs.  Additionally, 
the  JCS’s  Joint  Requirement  Oversight  Council  review  and  validation  of  those 
requirements  documents  did  not  focus  the  primary  requirements  on  the 
accomplishment  of  critical  aircraft  missions. 

These  conditions  occurred  because  JCS  guidance  did  not  specifically  require  the 
CNO  staff  and  the  Joint  Requirement  Oversight  Council  (JROC)  to  make  sure  the 
requirements  document  captures  system  characteristics  that  are  most  critical 
to  meeting  mission  requirements  as  primary  requirements.  As  a  result,  the 
weapon  system  sponsors  concurred  with  P-8A  and  the  E-2D  waiver  requests 
without  giving  the  JCS  the  opportunity  to  assess  the  effect  of  those  waivers  on 
the  military  usefulness  of  the  aircraft.  Additionally,  because  the  requirements 
documents  did  not  capture  the  most  critical  system  characteristics  as  primary 
requirements,  program  managers  accepted  production  units  with  diminished 
mission  effectiveness.  For  example,  at  the  time  of  the  final  production  decision, 
the  program  manager  for  the  P-8A  Poseidon  had  accepted  13  aircraft  for 
$2.6  billion  that  could  not  fully  perform  primary  missions  for  Anti-Submarine 
Warfare,  Anti-Surface  Warfare,  and  Intelligence  gathering. 


Weapon  Sponsors  Did  Not  Fully  Define  Critical  System 
Characteristics  as  Primary  Requirements 

Weapon  system  sponsors  within  the  Office  of  the  CNO  wrote  system  requirements 
documents  for  the  P-8A  and  E-2D  that  did  not  fully  define  system  characteristics 
associated  with  the  most  critical  aircraft  missions,  as  primary  requirements. 

The  JCIDS  ManuaF^  states  that  the  sponsor  designates  "appropriate"  system 
characteristics  as  primary  and  secondary  requirements;  however,  the  Manual  does 
not  emphasize  a  sponsor’s  responsibility  to  make  certain  that  those  system 


The  Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System,  January  19,  2012. 
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characteristics  most  critical  to  meeting  mission  requirements  are  captured  as 
primary  requirements.  We  are  recommending  the  Vice  Chairman,  JCS,  modify 
the  JCIDS  Manual  to  require  sponsors  to  define  those  system  characteristics 
most  critical  for  providing  an  effective  military  capability  and  designate  them  as 
primary  requirements  when  writing  system  requirements  documents.  The  sections 
below  provide  details  on  shortfalls  in  defining  primary  requirements  for  the 
P-8A  Poseidon  and  E-2D  ARE  aircrafts. 

P-8A  Poseidon  Aircraft 

The  Deputy  CNO  for  Air  Warfare  (N98),  the  weapon  system  sponsor  for  the 
P-8A,  designated  system  capability  requirements  directly  linked  to  the  aircraft’s 
primary  missions  to  secondary  or  lower  requirement  status.  CNO  N98  staff  should 
have  designated  performance  of  anti-submarine  warfare;  anti-surface  warfare; 
and  intelligence,  surveillance,  and  reconnaissance  missions  as  critical  system 
characteristics  rather  than  secondary  or  lower  level  requirements. 

The  systems  requirements  document  states  the  primary  reason  the  Navy  invested 
in  the  P-8A  aircraft  was  to  fill  a  capability  gap^^  to  find,  track,  and  destroy  or 
neutralize  hostile  submarines  to  support  the  anti-submarine  warfare  mission. 

The  document  also  identified  anti-surface  warfare  and  intelligence,  surveillance, 
and  reconnaissance  as  primary  mission  areas.  Further,  it  stated  the  P-8A  should 
have  the  capability  to  search  for,  detect,  identify,  track,  target,  engage,  and  provide 
damage  assessment  of  maritime  targets. 

The  capabilities  the  CNO  N98  staff  defined  as  primary  requirements  included 
basic  aircraft  performance  characteristics  such  as  range  and  cargo  capacity. 

For  example,  the  P-8A  must  be  able  to  fly  a  radius  of  1,200  nautical  miles  while 
carrying  10,000  pounds  of  sonobuoys  and  torpedoes  and  maintaining  a  minimum 
line-of-sight  range  for  sensors  and  communications. 

E-2D  Advanced  Hawkeye  Aircraft 

The  Deputy  CNO  for  Information  Dominance  (N2/N6],  the  weapon  system 
sponsor  for  the  E-2D,  designated  a  majority  of  the  radar  detection  capability 
to  "other  system  characteristics,"  in  the  requirements  document.  A  system 
characteristic  is  a  third  level  requirement  that  is  below  primary  and 
secondary  requirements. 


Capability  gaps  are  defined  as  the  difference  between  current  capabilities  and  those  needed  to  perform 
required  missions. 
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The  requirements  document  for  the  E-2D^'‘  explains  that  the  airborne  early  warning 
surveillance  and  command  and  control  systems,  which  the  E-2D  uses,  provide 
strike  group  commanders  with  real-time  threat  warnings  and  tactical  analysis  to 
support  control  of  assigned  response  forces.  The  document  further  states  that,  in 
conducting  the  airborne  early  warning  mission,  E-2D  crews  must  perform  the  "core 
functions”  that  detect,  track,  classify  and  identify  contacts  in  the  air  and  on  the 
sea.  Radar  detection  of  air  and  sea  contacts  is  a  core  function  that  the  crews  must 
perform  to  meet  the  airborne  early  warning  mission. 


However,  the  requirements  document  only  designates  the  ability  to  detect 
air  targets  without  electronic  attack  [jamming]  as  a  primary  requirement. 

Requirements  to  detect  air  targets  with  jamming  and  to 
^  detect  targets  on  the  surface  of  the  sea  are  designated  as 
third-level  requirements.  As  previously  explained,  the 
requirements  document  states  that  detection  of  air  and 
sea  contacts  is  a  "core  function"  that  E-2D  crews  must 
perform  to  meet  the  aircraft’s  airborne  early  warning 
mission.  Therefore,  CNO  N2/N6  staff  should  have 
designated  the  abilities  to  perform  radar  detection  of 
air  and  surface  threats  as  primary  rather  than  lower 
level  requirements. 


Requirements 
to  detect  air 
targets  with  jamming 
and  to  detect  targets 
on  the  surface  of  the 
sea  are  designated 
as  third-level 
requirements 


Review  of  Requirements  Documents  Needs  to  Focus  on 
Characteristics  Most  Critical  to  Accomplish  Missions 

The  JROC  review  and  validation  of  the  P-8A  and  E-2D  requirements  documents 
did  not  include  a  determination  that  the  requirements  documents  identified  those 
system  characteristics  most  critical  to  accomplish  primary  aircraft  missions 
as  primary  requirements.  This  did  not  occur  because  the  JCIDS  Manual  did 
not  emphasize  such  a  review.  For  example,  the  P-8A  anti-submarine  warfare; 
anti-surface  warfare;  and  intelligence,  surveillance,  and  reconnaissance  critical 
characteristics  were  not  captured.  For  the  E-2D,  radar  detection  of  air  and  sea 
threats,  a  critical  system  attribute,  was  not  captured. 

In  a  memorandum,^^  the  Director,  Operational  Test  and  Evaluation  notified  the 
Vice  Chairman,  JCS  that  the  requirements  document  for  the  P-8A  did  not  capture  as 
primary  requirements  those  characteristics  critical  for  mission  effectiveness.  On 
October  22,  2013,  the  Vice  Chairman,  JCS  responded  to  the  Director,  Operational  Test 
and  Evaluation,  stating  that  he  shared  the  Director’s  concerns  regarding  the 


"E-2D  Advanced  Hawkeye  Capability  Development  Document  (CDD)  for  Milestone  C,"  September  15,  2008. 
"P8-A  Poseidon  Multi-Mission  Maritime  Aircraft  Increment  1  Key  Performance  Parameters,"  September  4,  2013. 
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selected  primary  requirements  for  the  P-8A  program  and  their  inability  to  effectively 
contribute  to  operational  effectiveness  assessments.  The  Vice  Chairman  further 
stated  that  “amplifying  and  clarifying"  language  would  be  added  to  the  2012  JCIDS 
Manual  as  part  of  an  ongoing  update. 

However,  on  August  27,  2014,  we  received  a  summary  of  the  27  “major  changes" 
the  JCS  planned  as  updates  to  the  JCIDS  Manual,  and  none  of  the  planned  changes 
directly  addressed  the  described  deficiencies  above  with  primary  and  secondary 
requirement  value  selection.  We  are  recommending  that  the  Vice  Chairman,  JCS 
modify  the  JCIDS  Manual  to  add  clarifying  language  to  make  sure  the  JROC  review 
of  requirements  documents  includes  verifying  that  weapon  system  sponsors  include 
characteristics  most  critical  to  mission  effectiveness  as  primary  requirements. 

Requirements  Documents  Need  Improvement  for 
Better  Visibility  of  Waiver  Approvals  and  Mission 
Effectiveness  Determinations 

Requirements  documents  that  do  not  establish,  as  primary  requirements,  those 
system  characteristics  most  critical  to  mission  accomplishments  result  in: 

•  the  JCS  not  having  visibility  over  the  approval  of  waiver  requests,  and 

•  the  Director,  Operational  Test  and  Evaluation  being  hindered  in 
determining  the  operational  effectiveness  of  weapon  systems. 

ReducedJoint  Chiefs  of  Staff  Visibility  of  Waiver  Approvals 

Sponsors  for  the  P-8A  and  E-2D  aircraft  programs  agreed  with  the  waiver  request 
without  a  requirement  to  determine  the  effect  those  waivers  would  have  on 
military  utility^^  because  they  did  not  affect  primary  requirements.  The  JROC 
would  have  required  a  review  to  evaluate  the  operational  risk  and  military  utility 
of  the  system  if  the  primary  requirement  thresholds  were  not  going  to  be  met. 

Specifically,  the  JCIDS  Manual  states  that  failure  of  a  system  to  meet  primary 
requirement  thresholds  brings  the  military  utility  of  the  system  into 
question  and  may  result  in  a  reevaluation  of  the  program  or  modification  to 
production  increments.  Additionally,  Chairman  of  the  Joint  Chiefs  of  Staff 
Instruction  3170. OlH^^  states  that  interaction  must  occur  between  Service  weapon 
system  sponsors  and  the  JCS,  when  there  are  changes  during  production  that 
require  changes  to  validated  primary  requirement  thresholds. 


A  system  designed  for  or  possessing  a  number  of  useful  or  practical  purposes  rather  than  a  single,  specialized  one. 
Chairman,  Joint  Chiefs  of  the  Staff  Instruction  3170. OlH,  "Joint  Capabilities  Integration  and  Development  System," 
January  10,  2012. 
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The  updated  "ready  for  signature”  draft  version  of  the  JCIDS  Manual,  dated 
November  7,  2014,  will  clarify  the  required  interaction  between  the  Service 
weapon  system  sponsor  and  the  JCS.  Specifically,  the  update  explains  that 
failure  of  a  system  to  meet  a  validated  primary  requirement  threshold  triggers  a 
review  by  the  validation  authority  and  evaluation  of  operational  risk  and  military 
utility  of  the  system  if  the  primary  requirement  threshold  is  not  met.  Both  the 
2012  JCIDS  Manual  and  the  planned  draft  update  state  that  the  validation  authority 
is  within  the  JCS  for  Acquisition  Category  I  programs,  such  as  the  P-8A  and 
E-2D  aircraft,  and  for  programs  designated  of  interest  to  the  Joint  Capability  Board 
or  the  Joint  Requirements  Oversight  Council  of  the  JCS. 

P-8A  Poseidon 

The  P-8A  PEO  requested  a  waiver  of  the  criterion,  no  uncorrected  flight 
deficiencies,  with  CNO  concurrence.  As  part  of  the  waiver  request,  the  program 
manager  listed  28  unresolved  flight  deficiency  reports.  The  reports  identified 
critical  deficiencies  that  affected  the  ability  of  the  aircraft  to  accomplish  its 
primary  missions  of  anti-submarine  warfare,  anti-surface  warfare,  and  intelligence, 
surveillance,  and  reconnaissance.  The  following  table  shows  the  28  flight 
deficiencies  the  program  manager  listed  in  his  waiver  request  and  the  P-8A  aircraft 
mission  areas  affected. 

(FOUO)  Table:  Break  out  of  Deficiencies  and  Potential  Mission  Impacts 


(FOUO)  Number  of 

Mission  Area  Deficiency  Summary  of  Potential  Mission  Impacts 

Affected  Reports 
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(FOUO)  Number  of 

Mission  Area  Deficiency  Summary  of  Potential  Mission  Impacts 

Affected  Reports 


E-2D  Advanced  Hawkeye 

(FOUO)  The  E-2D  PEO  also  requested  a  waiver  of  the  criterion,  no  uncorrected 
flight  deficiencies,  which  the  CNO  staff  granted.  As  part  of  the  waiver  request, 
the  program  manager  listed  one  unresolved  flight  deficiency  report  relating  to 
radar  detection. 

We  recommend  that  ASN(RDA)  update 
SECNAVINST  5000. 2. E  to  require  sponsors  to  certify  to  the  JCS  that  approved 
waivers  do  not  adversely  affect  primary  system  requirements  or,  if  a  proposed 
waiver  does  adversely  affect  primary  system  requirements,  that  the  system 
continues  to  have  military  utility  as  described  in  the  JCIDS  Manual  before 
approving  the  waiver. 

Shortfalls  in  Defining  Primary  Requirements  for 
P-8A  Poseidon  Aircraft  Hindered  Determination  of 
System  Mission  Effectiveness 

On  September  4,  2013,  the  Director,  Operational  Test  and  Evaluation  alerted 
the  Vice  Chairman,  JCS  and  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  about  shortfalls  in  defining  primary  requirements  for 
the  P-8A.  The  Director  stated  that  the  lack  of  primary  requirements  related 
directly  to  mission  effectiveness  creates  a  disconnect  between  the  determination  of 
operational  effectiveness  in  test  reports  and  the  primary  requirement  compliance 
assessments  that  drive  program  reviews  throughout  system  development.  The 
Director  also  stated  the  aircraft  was  "fully  compliant  with  KPP/KSA  [primary  and 
secondary  requirements]  thresholds,  while  having  significant  shortfalls  in  mission 
effectiveness."  The  Director  stated  this  was  an  indication  that  the  most  essential 
operational  requirements  were  focused  too  narrowly. 
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Director,  Operational  Test  and  Evaluation  lOT&E  report^®  P-8A  primary 
requirements  thresholds  were  limited  to  mission  support  characteristics, 
such  as  basic  aircraft  performance,  availability,  survivability,  force  protection, 
and  information  exchange  characteristics.  The  primary 
requirements  did  not  include  specific  performance 
thresholds  for  the  critical  anti-submarine  warfare, 
anti-surface  warfare,  or  intelligence,  surveillance,  and 
reconnaissance  mission  tasks  that  define  operational 
effectiveness.  The  Director  stated  that,  in  an 
extreme  case,  the  contractor  could  deliver  an  aircraft 
that  met  all  the  primary  requirements  but  has  no 
mission  capabilities.  The  Director  also  explained 
that  primary  requirements  should  provide  standards 
for  determining  mission  accomplishment  and  summarize 
the  reasons  for  procuring  a  system.  Because  the  requirements  documents  did  not 
capture  the  most  critical  system  characteristics  as  primary  requirements,  program 
managers  accepted  production  units  with  diminished  mission  effectiveness.  For 
example,  at  the  time  of  the  final  production  decision,  the  program  manager  for 
the  P-8A  Poseidon  had  accepted  13  aircraft  for  $2.6  billion  that  could  not  fully 
perform  primary  missions  for  anti-submarine  warfare,  anti-surface  warfare,  and 
intelligence  gathering. 


Primary 
iquiremen 
ould  proviae 
standards  for 
determining  mission 
accomplishment  and 
summarize  the  reasons 
for  procuring  a 
system. 


Management  Action  Started  During  the  Audit 

In  response  to  this  audit,  the  JCS  plans  to  update  the  JCIDS  Manual  with  clarifying 
language  on  writing  and  reviewing  system  requirements  documents  to  require: 

•  sponsors  to  include  system  characteristics  most  critical  to  mission 
effectiveness  as  primary  requirements;  and 

•  a  JROC  review  to  make  sure  the  system  characteristics  most  critical  to 
meeting  mission  requirements  are  captured  as  primary  requirements. 

Management  Comments  on  the  Finding 
and  Our  Response 

The  Deputy  Department  of  the  Navy  Test  and  Evaluation  Executive,  responding  for 
the  Assistant  Secretary  of  the  Navy  for  Research,  Development,  and  Acquisition, 
provided  the  following  comments  on  the  finding.  For  the  full  text  of  the  Deputy's 
comments,  see  the  Management  Comments  sections  of  the  report. 


28  "p_gA  Poseidon  Multi-Mission  Maritime  Aircraft  Initial  Operational  Test  and  Evaluation  Report,"  October  2013. 
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Department  of  the  Navy  Comments 

The  Deputy  disagreed  with  the  finding  stating  that: 

•  the  finding  implies  the  only  way  the  program  manager  and  PEOs  are 
informed  of  a  system’s  required  capabilities  and  determines  whether 
the  system  is  ready  to  proceed  to  operational  testing  is  through  reading 
a  detailed  requirements  document.  This  is  incorrect;  the  requirement 
document  is  not  the  only  vehicle  for  conveying  programmatic  information 
to  the  PEG  and  program  manager. 

•  resource  sponsors,  program  managers,  and  PEOs  are  in  daily 
communications  discussing  risk,  cost,  performance,  and  schedule. 

•  numerous  technical  and  fiscal  realities,  as  well  as  actual  system 
capabilities,  are  included  in  the  decision  to  test  or  field  the  system  with 
a  specific  level  of  performance. 

•  the  existing  SECNAVINST  5000. 2E,  paragraph  4.6. 2. 2  states  that  when 
PEOs  certify  readiness  for  operational  test  with  exceptions,  they  must 
send  a  message  to  CNO  N84  and  provide  information  copies  to  the 
program  sponsor  for  their  formal  concurrence,  and  to  the  ASN(RD&A), 
and  Commander,  Operational  Test  and  Evaluation  Force. 

•  this  policy  makes  certain  that  the  PEO  delegated  Title  10  responsibility 
from  the  CNO  to  develop  requirements  for  systems  that  will  equip  the 
fleet  have  executed  that  responsibility  and  determined  the  system 
capabilities  are  adequate  to  proceed  to  operational  testing.  The  Navy 
views  this  as  adequate  oversight  and  that  adding  the  requirement  for  the 
JROC  to  approval  waivers  would  be  a  significant  administrative  burden. 

•  the  Fleet’s  most  urgent  need  for  the  P-8A  platform  was  for  the 
anti-submarine  warfare  mission  conducted  by  the  aging  P-3C  Orion. 

The  mission  systems  for  the  anti-submarine  warfare  and  other  missions 
were  derivatives  of  mature  technologies  that  were  successfully  performed 
the  same  missions  on  the  P-3C.  The  requirements  were  logically  written 
to  define  the  needed  replacement  platform,  and  platform  performance  was 
appropriately  designated  as  one  of  the  most  critical  characteristics  of  the 
P-8A  as  a  platform  replacement  program. 

Our  Response 

According  to  the  JCIDS  Manual,  the  purpose  of  the  JCIDS  process  and  resulting 
system  requirements  documents  is  to  define  authoritative,  measurable,  and  testable 
requirements  across  one  or  more  increments  of  a  materiel  capability  solution. 

These  requirements  must  be  properly  defined  to  allow  resources  sponsors,  PEOs, 
and  program  managers  to  have  meaningful  discussions  of  risk,  cost,  performance, 
and  schedule.  Additionally,  as  further  discussed  in  our  response  to  the  Deputy’s 
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comments  on  Recommendation  B.2,  the  failure  of  a  system  to  meet  a  validated 
primary  requirement  threshold  triggers  a  review  by  the  validation  authority  and 
evaluation  of  operational  risk  and  military  utility  of  the  system  if  the  primary 
requirement  threshold  is  not  met.  The  updated  JCIDS  ManuaP^  states  that  the 
validation  authority  is  within  the  JCS  for  Acquisition  Category  1  programs,  such  as 
the  P-8A  and  E-2D  Aircraft  we  reviewed.  Further,  Chairman  of  the  Joint  Chiefs  of 
the  Staff  Instruction  3170.01H^°  states  that  interaction  must  occur  between  Service 
weapon  system  sponsors  and  the  JCS,  when  there  are  changes  during  production 
that  require  changes  to  validated  primary  requirement  thresholds. 

We  recognize  that  the  Navy  designed  the  P-8A  Poseidon  to  replace  the  aging 
P-3C  Orion  aircraft.  However,  the  P-8A  had  25  critical  flight  deficiencies  that 
affected  the  ability  of  the  aircraft  to  accomplish  its  primary  anti-submarine 
warfare  mission.  As  discussed  in  the  finding,  the  Director,  Operational  Test  and 
Evaluation  noted  that  the  lack  of  primary  requirements  relating  directly  to  mission 
effectiveness  creates  a  disconnect  between  the  determination  of  operational 
effectiveness  in  test  reporting  and  the  primary  requirements  compliance 
assessment  that  drive  program  reviews  through  system  development. 

Recommendations,  Management  Comments, 
and  Our  Response 

Revised  Recommendation 

As  a  result  of  the  Deputy  Department  of  the  Navy  Test  and  Evaluation 
Executive  comments  to  recommendations  in  a  draft  of  this  report,  we  revised 
Recommendation  B.2  to  clarify  the  need  for  acquisition  programs  designated  as 
Acquisition  Category  I,  or  of  interest  to  the  Joint  Capability  Board  or  the  JROC,  to 
certify  to  the  Vice  Chairman,  JCS  the  impact  a  waiver  of  certification  criteria  has  on 
meeting  primary  requirements. 

Recommendation  B.l 

We  recommend  that  Vice  Chairman,  Joint  Chiefs  of  Staff,  modify  the  Joint 
Capabilities  Integration  and  Development  System  Manual  to  add  clarifying 
language  regarding  the  need  for: 

a.  sponsors  to  define  those  system  characteristics  that  are  most  critical  for 
providing  an  effective  military  capability  and  designate  them  as  primary 
requirements  when  writing  requirements  documents;  and 


Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System,  February  12,  2015. 

Chairman  of  the  Joint  Chiefs  of  the  Staff  Instruction  3170. OlH,  "Joint  Capabilities  Integration  and  Development  System," 
January  10,  2012. 
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b.  Joint  Requirements  Oversight  Council  review  to  make  sure  the  system 
characteristics  most  critical  to  mission  effectiveness  are  captured  as 
primary  requirements. 

Vice  Chairman,  Joint  Chiefs  of  Staff  Comments 

The  Director;  Joint  Staff,  responding  for  the  Vice  Chairman,  JCS,  agreed,  stating 
in  February  2015  he  signed  the  updated  JCIDS  Manual  that  includes  clarifying 
language  to  address  the  recommendation. 

Our  Response 

The  Director,  Joint  Staff  addressed  the  specifics  of  the  recommendation,  and  no 
further  comments  are  required. 

Deputy  Department  of  the  Navy  Test  and  Evaluation 
Executive  [N84C]  Comments 

Although  not  required  to  comment,  the  Test  and  Evaluation  Executive  agreed  with 
the  recommendation  and  stated  that  the  information  was  incorporated  in  the 
updated  JCIDS  Manual  and  there  is  no  impact  to  existing  processes  or  Navy  policy. 

Recommendation  B.2 

We  recommend  that  Assistant  Secretary  of  the  Navy  for  Research,  Development 
and  Acquisition  update  Secretary  of  the  Navy  Instruction  5000. 2. E,  “Department 
of  the  Navy  Implementation  and  Operation  of  the  Defense  Acquisition  System  and 
the  Joint  Capabilities  Integration  and  Development  System,"  September  1,  2011, 
to  require  sponsors  of  acquisition  programs  designated  as  Acquisition  Category  1, 
or  of  interest  to  the  Joint  Capability  Board  or  the  Joint  Requirements  Oversight 
Council,  to  certify  to  the  Vice  Chairman,  Joint  Chiefs  of  Staff  that: 

a.  approved  waivers  of  certification  criteria  do  not  adversely  impact 
primary  system  requirements;  or 

b.  if  a  proposed  waiver  of  certification  criteria,  such  as  the  criteria  to  meet 
system  performance  requirements  in  the  test  and  evaluation  master  plan, 
does  adversely  impact  primary  system  requirements,  that  the  system 
continues  to  have  military  utility  as  described  in  the  “Manual  for  the 
Operation  of  the  Joint  Capabilities  Integration  and  Development  System," 
January  10,  2012  before  approving  the  waiver. 
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Deputy  Department  of  the  Navy  Test  and  Evaluation 
Executive  (N84C)  Comments 

The  Deputy  Department  of  the  Navy  Test  and  Evaluation  Executive  responding 
for  the  ASN(RD&A)  disagreed,  stating  that  adding  a  requirement  for  a  waiver  to 
be  approved  by  JROC  would  also  drive  many  flag-level  briefings  before  the  waiver 
would  even  leave  the  Navy  to  start  Joint  Staffing.  Specifically,  JROC  has  Title  10 
authority  over  requirements,  not  acquisition  implementation  of  them.  After 
JROC  validates  the  requirements,  it  becomes  CNO’s  responsibility  to  make  sure 
they  are  met  and  ASN(RD&A)  responsibility  to  acquire  the  appropriate  material 
solution.  The  Deputy  further  stated  that  the  PEOs,  in  consultation  with  CNO  staff, 
are  required  to  assess  whether  a  system  shortfall  impacts  meeting  system 
requirements  when  they  certify  readiness  to  enter  operational  test. 

As  the  Deputy  previously  stated  the  existing  SECNAVINST  5000. 2E,  paragraph  4.6. 2. 2 
states  that  when  PEOs  is  certify  readiness  for  operational  test  with  exceptions 
they  must  send  a  message  to  CNO  N84  and  provide  information  copies  to  the 
program  sponsor  for  their  formal  concurrence,  and  ASN(RD&A)  and  Commander, 
Operational  Test  and  Evaluation  Force.  The  Deputy  further  stated  that  this  policy 
makes  certain  that  the  flag  officers  delegated  Title  10  responsibility  from  the  CNO 
to  develop  requirements  for  systems  have  executed  that  responsibility  to  determine 
that  the  system  capabilities  are  adequate  to  proceed  to  operational  testing. 

Furthermore,  the  Deputy  stated  that  the  current  waiver  and  deferral  process  has 
historically  worked  well  for  all  PEO,  Air  Anti-Submarine  Warfare,  Assault  and 
Special  Missions  programs.  The  recommendation  would  effectively  require  major 
defense  acquisition  programs  to  regularly  elevate  discussions  of  narrowly  scoped 
individual  deficiencies  with  minimal  impact  on  top-level  system  performance  to  the 
JROC  or  JCS  and  would  create  a  significant  administrative  burden. 

Our  Response 

The  Deputy  did  not  address  the  specifics  of  the  recommendation.  We  understand 
that  it  will  take  additional  time  to  certify  to  the  Vice  Chairman,  JCS  the  impact 
waived  criteria  has  on  primary  requirements.  However,  according  to  Force 
Structure,  Resources  and  Assessment  Directorate  (J-8)  staff,  the  JCS  would  be 
flexible  and  try  to  make  the  review  as  short  as  possible.  J-8  staff  further  stated 
that  being  flexible  is  not  a  problem  as  long  as  the  Navy  recognizes  waiver-related 
deficiencies  early  and  communications  to  the  JCS. 
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Additionally,  the  Deputy  stated  that  the  JROC  has  Title  10  authority  over 
requirements,  not  the  acquisition  implementation  of  them.  However,  the  updated 
JCIDS  process  requires  JCS  involvement  when  the  program  manager  for  an 
Acquisition  Category  1  or  special  interest  program  no  longer  expects  the  system  to 
meet  its  primary  mission  requirements. 

The  updated  JCIDS  Manual  explains  that  failure  of  a  system  to  meet  a  validated 
primary  requirement  threshold  triggers  a  review  by  the  validation  authority  and 
evaluation  of  operational  risk  and  military  utility  of  the  system  if  the  primary 
requirement  threshold  is  not  met.  The  updated  Manual  then  explains  that  the 
validation  authority  is  within  the  JCS  for  Acquisition  Category  1  programs,  such  as 
the  P-8A  and  E-2D  Aircraft,  and  for  programs  designated  of  interest  to  the  Joint 
Capability  Board  or  the  Joint  Requirements  Oversight  Council  of  the  JCS. 

Therefore,  we  request  the  Deputy  provide  additional  comments  on  the  final  report 
explaining  how  he  would  comply  with  JCIDS  Manual  without  informing  JCS  when 
the  PEO  programs  do  not  meet  the  primary  requirements. 
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Scope  and  Methodology 

We  conducted  this  performance  audit  from  June  2014  through  March  2015  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained  provides  a 
reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 

We  interviewed  key  personnel  and  performed  fieldwork  at  the 
following  organizations: 

•  Joint  Staff  J-8  Force  Structure,  Resource,  and  Assessment,  Washington,  D.C. 

•  Director,  Operational  Test  and  Evaluation,  Washington,  D.C. 

•  Assistant  Secretary  of  the  Navy  for  Research,  Development,  and 
Acquisition,  Washington,  D.C. 

•  Director,  Innovation,  Test,  and  Evaluation,  and  Technology  Requirements, 
Washington,  D.C. 

•  Commander,  Operational  Test  and  Evaluation  Force,  Norfolk,  Virginia 

•  P-8A  Poseidon  Aircraft  Program  Office,  Patuxent,  Maryland 

•  E-2D  Advanced  Hawkeye  Aircraft  Program  Office,  Patuxent,  Maryland 

•  Distributed  Targeting  System  Program  Office,  Patuxent,  Maryland 

We  collected,  reviewed,  and  analyzed  documents  dated  October  2009  through 
November  2014.  We  reviewed  requirements  documents,  test  and  evaluation  plans 
and  reports,  operational  test  readiness  review  checklists  and  briefing  charts,  and 
program  certification  messages  that  requested  waivers  and  deferrals  to  determine 
whether  NAVAIR  staff  adequately  justified,  reviewed,  and  approved  the  waiver  and 
deferral  of  operational  testing  requirements  for  acquisition  of  NAVAIR  systems. 

Additionally,  we  reviewed  program  planning  and  reporting  documents 
and  compared  them  to  the  policies  and  guidance  in  the  following  DoD  and 
Navy  issuances. 

•  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3170. OlH,  "Joint 
Capabilities  Integration  and  Development  System,"  January  10,  2012 

•  "Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and 
Development  System,”  January  19,  2012 

•  Defense  Acquisition  Guidebook,  September  16,  2013 
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•  Secretary  of  the  Navy  Instruction  5000. 2E,  "Department  of  the  Navy 
Implementation  and  Operation  of  the  Defense  Acquisition  System  and  the 
Joint  Capabilities  Integration  and  Development  System,"  September  1,  2011 

•  Secretary  of  the  Navy  Manual  5000.2,  "Department  of  the  Navy 
Acquisition  and  Capabilities  Guidebook,"  May  9,  2012 

Selection  of  Programs  to  Review 

We  obtained  a  query  from  Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition  Information  Systems  database  to  identify  Navy 
weapon  systems  acquisition  programs  that  received  a  final  production  decision 
from  April  14,  2012,  through  April  14,  2014.  We  identified  three  NAVAIR  programs 
(P-8A  Poseidon,  E-2D  Advanced  Hawkeye,  and  DTS]  and  conducted  a  100-percent 
review  of  those  programs  to  evaluate  the  command’s  management  of  the  waiver 
and  deferral  process. 

Use  of  Computer-Processed  Data 

We  did  not  use  computer-processed  data  to  perform  this  audit. 

Prior  Coverage 

During  the  last  5  years,  the  Department  of  Defense  Inspector  General  (DoD  IG) 
issued  one  report  discussing  operational  test  waivers  and  deferrals.  Unrestricted 
DoD  IG  reports  can  be  accessed  at  http://www.dodig.mil/pubs/index.cfm. 

DoD  IG 

(FOUO)  Report  No.  DODIG-2013-088,  "The  Navy  P-8A  Poseidon  Aircraft  Needs 
Additional  Critical  Testing  Before  the  Full-Rate  Production  Decision,"  June  10,  2013. 
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Joint  Chiefs  of  Staff  Comments 


JOINT  STAFF  ACTION  PROCESSING  FORM 

CLASSIFICATION  UNCLASSIFIED/ /FOR  OFFICIAL  USE  ONLY 


EXTERNAL  SUSPENSE  13  Apr  15 


STAFF  SUSPENSE  7  Apr  15 


ACTION  NUMBER  15-01 107 


TO  DJS 


THRU  VDJS 


J  A.. 


SUBJECT  /0-D0DIG-FA2/  Naval  Air  Systems  Command  Needs  to  Improve  Management  of  Waiver  Requests 
(Project  No.  D2014-D000AE- 164.000) 


EXECUTIVE  SUMMARY 

1 .  Purpose.  Provide  the  Inspector  General  of  the  Department  of  Defense  (DoD  IG)  with  Joint 
Steiff  review  and  comment  on  the  subject  draft  report.  ^ 

2.  Issue.  The  DoD  IG  draft  report  provides  recommendations  for  Joint  Staff  action  on  pages 
ii,  15,  17,  and  21. 

3.  Bottom  Line.  The  Joint  Staff  should  concur  with  the  DoD  IG  draft  report  as  written.  All 
actions  recommended  in  the  report  have  been  completed. 

4.  Background 

a.  DoD  IG  evaluated  the  Naval  Air  Systems  Command’s  management  of  waivers  and 
deferrals  from  operational  test  requirements  for  weapons  systems.  DoD  IG  recommends  areas 
to  improve  language  in  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS) 
manual. 

b.  The  February  2015  JCIDS  manual,  signed  by  VCJCS,  includes  clarifying  language  that 
addresses  the  DoD  IG  audit  team’s  recommendations. 

5.  Recommendation .  DJS  approve  concurrence  as  stated  in  the  bottom  line. 


^  y  Disapprove 


Approve 

DAVID  L.  GOLDFEIN,  Lt  Gen,  USAF 
Director,  Joint  Staff 
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Management  Comments 


Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition  Comments 


DEPARTMENT  OF  THE  NAVY 

OFFICE  OF  THE  CHIEF  OF  NAVAL  OPERATIONS 
2000  NAVY  PENTAGON 
WASHINGTON,  D.C.  20350-2000 

IN  REPLY  REFER  TO 

14  April  2015 


MEMORANDUM  FOR  INSPECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 

SUBJECT:  Navy  Response  to  DODIG  Draft  Audit  Report,  “NAVAL  AIR  SYSTEMS 
COMMAND  NEEDS  TO  IMPROVE  MANAGEMENT  OF  WAIVER 
REQUESTS”,  D2014-D000AE-I64.000,  Dated  12  Mar  2015 

The  Navy  has  reviewed  the  subject  report  and  partially  concurs  with  the  findings 
and  recommendations.  NAVAIRSYSCOM  (PMA  290,  324  and  265)  and  OPNAV  N84 
assisted  the  DODIG  during  the  conduct  of  this  audit,  and  the  report  and  responses  were 
socialized  within  the  offices  of  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition),  the  Chief  of  Naval  Operations  and  NAVAIRSYSCOM. 
Our  response  to  your  draft  report  is  enclosed. 

Responsibility  for  implementing  the  proposed  changes  that  Navy  concurs  with  is 
the  responsibility  of  OPNAV  N84.  If  you  have  any  questions,  please  contact 


Carroll  P.  Quade 

Deputy  Department  of  the  Navy 

Test  and  Evaluation  Executive  (N84C) 
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Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition  Comments  (cont'd) 


NAVY  RESPONSE  TO 

DODIG  DRAFT  AUDIT  REPORT,  “NAVAL  AIR  SYSTEMS  COMMAND  NEEDS  TO 
IMPROVE  MANAGEMENT  OF  WAIVER  REQUESTS”, 
D20I4-D000AE-I64.000,  DATED  12  MARCH  2015 


Finding  A:  Naval  Air  Systems  Command  Needs  to  Improve  Management  of  Waiver 
Requests 

NAVAIR  Program  Managers  (PMs)  did  not  fully  implement  Navy  policy  to  request 
waivers  and  certify  program  readiness  for  Initial  Operational  Test  and  Evaluation 
(lOT&E)  on  the  P-8A  aircraft.  Distributed  Targeting  System  (DTS),  and  E-2D  aircraft 
programs.  Specifically,  PMs  did  not: 

•  request  waivers  for  the  P-8A  and  DTS  when  those  programs  did  not  meet  all 
lOT&E  certification  criteria;  and 

•  clearly  document  what  was  done  to  meet  each  of  the  certification  criteria  for 
the  E-2D. 

These  conditions  occurred  because  Navy  policy  did  not  clearly  and  concisely  state  that 
PMs  must  request  a  waiver  whenever  the  program  did  not  meet  all  criteria  required  to 
enter  lOT&E.  Because  program  managers  did  not  request  waivers,  the  Program 
Executive  Officer  (PEO)  could  not  effectively  consider  program  readiness  for  entering 
the  lOT&E  test  phase.  As  a  result,  the  P-8  A  and  the  DTS  program  entered  and 
completed  lOT&E  with  unresolved  deficiencies  affecting  threat  detection, 
maneuverability,  information  exchange,  and  reliability.  These  deficiencies  diminished 
the  ability  of  the  P-8A  aircraft  and  the  DTS  to  perform  their  primary  missions. 

QPNAV  Response: 

N84:  Do  not  concur 

This  finding  states  that  because  PMs  did  not  request  waivers,  the  PEO  could  not  effectively 
consider  program  readiness  for  entering  the  lOT&E  test  phase.  This  implies  that  the  only 
information  discussed  at  the  Operational  Test  Readiness  Review  (OTRR)  is  regarding 
certification  criteria  not  met,  and  that  the  only  information  provided  to  the  PEO  to  support  his  or 
her  decision  is  during  the  OTRR.  This  is  incorrect. 

The  OTRR  is  a  culmination  of  many  months  of  effort  by  the  program  office.  The  program  Test 
and  Evaluation  Working  Integrated  Product  Team  (T&E  WIPT),  which  includes  resource 
sponsors,  DOT&E  and  DASD  (DT&E),  N842,  COTE,  logistics,  technical  test  experts,  training 
and  other  Subject  Matter  Experts  (SMEs),  determines  the  status  of  each  of  the  20  items  of  the 
OTRR  certification  criteria.  The  WIPT  also  determines  the  level  of  detail  that  each  of  the  criteria 
should  be  discussed  at  the  OTRR.  Multiple  iterations  of  the  brief  are  developed  and  reviewed 
before  presenting  to  the  PM  and  the  Deputy  PEO  before  proceeding  to  the  actual  OTRR.  There 
are  many  layers  of  expert  review  to  ensure  that  PEO  is  provided  objective  data  to  make  an 
informed  decision  at  the  OTRR. 
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Management  Comments 


Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition  Comments  (cont'd) 


NAVY  RESPONSE  TO 

DODIG  DRAFT  AUDIT  REPORT,  “NAVAL  AIR  SYSTEMS  COMMAND  NEEDS  TO 
IMPROVE  MANAGEMENT  OF  WAIVER  REQUESTS”, 
D20I4-D000AE-I64.000,  DATED  12  MARCH  2015 

The  existing  SECNAVINST  5000.2  para  4.6.2  states  “The  SYSCOM  commander,  PEO,  and 
DRPM  shall,  unless  otherwise  directed  by  ASN(RD&A)  for  programs  on  the  OSD  T&E  oversight 
list,  make  one  of  the  following  certifications”.  The  next  two  paragraphs  of  the  instruction  detail  the 
two  certifications,  and  state  that  the  PEO  shall  certify,  by  naval  message,  that  a  system  is  either 
certified  to  proceed  to  Operational  Testing  (OT)  with  no  exceptions,  or  certified  to  proceed  to  OT 
with  T&E  exceptions. 

N84  acknowledges  that  the  current  SECNAV  5000.2E  policy  could  be  updated  to  provide  more 
clear  guidance  on  leadership  expectations  of  OTRR  processes. 

NAY  AIR  Response: 

PMA-290:  Do  not  concur 

The  P-8A  OTRR  provided  the  PEO  an  extremely  detailed  briefing  with  several 
supporting  reports  for  which  to  base  the  readiness  for  test  decision  upon  including  each 
draft  wavier  and  deferral.  Additionally,  the  OTRR  included  representatives  from  all 
program  stakeholders  and  SMEs  who  carefully  reviewed  the  information  and  advised  the 
PEO  on  whether  criteria  were  satisfactorily  met  including  the  draft  waivers  and  deferrals. 

PMA-265:  Do  not  concur 

PMA-265  has  reviewed  the  report  and  believes  that  while  the  report  is  not  inaccurate,  the 
reliability  statements  on  DTS  are  misleading.  At  the  time  of  OTRR  (14  June  2012),  there 
were  1 16.9  flight  hours  between  operational  mission  failures  with  all  reliability  growth 
curves  on  track  to  reach  the  minimum  requirement  of  130  flight  hours  before  initial 
operational  capability.  DTS  reached  the  130  flight  hours  between  operational  mission 
failures  requirement  28  days  later  during  OT  on  12  July  2012.  LAW  SECNAVINST 
5000.2E,  Certification  of  Readiness  for  Operational  Testing,  para  4.6.1  states  the 
requirement  of:  T&E  results  indicate  performance  thresholds  identified  in  the  TEMP 
have  been  satisfied  or  are  projected  to  meet  system  maturity  for  the  CDD  and  CPD,  as 
appropriate.  As  previously  stated,  the  DTS  was  on  a  growth  curve  for  reliability  above 
the  requirements  identified  in  the  Test  and  Evaluation  Master  Plan  (TEMP)  and  as  such, 
exceeded  the  requirement  to  enter  into  the  lOT&E  phase. 

PMA-231:  Do  not  concur 

PMA-23 1  has  reviewed  the  report  and  believes  that  while  the  report  is  not  inaccurate,  the  PM  did 
document  what  was  done  to  meet  each  of  the  lOT&E  certification  criteria  for  the  E-2D  aircraft 
program. 

As  documented  in  the  DoDIG  report,  charts  contained  in  the  12  January  2012  briefing  “E-2D 
Road  to  lOT&E  Deep  Dive”  were  given  to  the  PEO  and  staff  and  certification  area  SMEs. 
Additionally,  the  DoDIG  report  acknowledges  that  information  contained  in  the  “E-2D  Road  to 
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Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition  Comments  (cont'd) 


NAVY  RESPONSE  TO 

DODIG  DRAFT  AUDIT  REPORT,  “NAVAL  AIR  SYSTEMS  COMMAND  NEEDS  TO 
IMPROVE  MANAGEMENT  OF  WAIVER  REQUESTS”, 
D20I4-D000AE-I64.000,  DATED  12  MARCH  2015 

lOT&E  Deep  Dive”  charts  was  provided  to  the  PEO  and  showed  evidence  of  how  the  program 
met  each  of  the  20  OTRR  certification  criteria.  Therefore,  the  chairperson  of  the  OTRR  was 
presented  with  the  certification  criteria  checklist  and  substantiating  evidence  showing  which 
certification  criteria  were  met  and  not  met. 

The  I  February  2012  OTRR  briefing  provided  a  checklist  summary  of  the  certification  criteria 
listed  in  SECNAVINST  5000.2E  since  the  chairperson  of  the  OTRR  was  already  provided  with 
the  substantiating  evidence  in  the  “E-2D  Road  to  lOT&E  Deep  Dive”  charts  from  12  January 
2012. 

Recommendation  A.l:  We  recommend  that  the  Assistant  Secretary  of  the  Navy  for  Research, 
Development,  and  Acquisition  and  the  Chief  of  Naval  Operations,  Director,  Innovation,  Test  and 
Evaluation,  and  Technology,  update: 

•  Secretary  of  the  Navy  Instruction  5000.2E,  “Department  of  the  Navy 
Implementation  and  Operation  of  the  Defense  Acquisition  System  and  the  Joint 
Capabilities  Integration  and  Development  System;”  and 

•  Secretary  of  the  Navy  Manual,  M-5000.2,  “Acquisition  and  Capabilities 
Guidebook,”  May  9,  2012,  Section  4.6,  “Certification  of  Readiness  for 
Operational  Testing,”  to: 


a.  Emphasize  that  PMs  must  request  waivers  whenever  they  do  not  meet  any 
of  the  20  criteria  the  Secretary  of  the  Navy  Instruction  5000.2E  requires 
programs  to  meet  to  certify  readiness  for  lOT&E;  and 

b.  Clarify  that  OTRR  briefings  to  stakeholder  groups  should  include  specific 
explanations  of  program  accomplishments  against  each  of  the  20 
certification  criteria  to  clearly  document  either  that  the  criteria  was  met  or 
a  waiver  or  deferral  request  was  coordinated  with  the  Chief  of  Naval 
Operations,  Director,  Innovation,  Test  and  Evaluation,  and  Technology; 
the  program  sponsors;  and  the  Commander,  Operational  Test  and 
Evaluation  Force. 


QPNAV  Response: 

N84:  Concur 

1 .  SECNAV  is  determining  the  best  way  to  promulgate  future  Navy  acquisition  policy 
(SECNAV  5000.2E).  While  leadership  considers  options,  the  Deputy  Department  of  the 
Navy  T&E  Executive  (N84C)  will  promulgate  interim  guidance  to  address  the  waiver  and 
deferral  process.  Specific  items  that  will  be  addressed: 
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Management  Comments 


Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition  Comments  (cont'd) 


NAVY  RESPONSE  TO 

DODIG  DRAFT  AUDIT  REPORT,  “NAVAL  AIR  SYSTEMS  COMMAND  NEEDS  TO 
IMPROVE  MANAGEMENT  OF  WAIVER  REQUESTS”, 
D20I4-D000AE-I64.000,  DATED  12  MARCH  2015 


a.  The  use  of  the  term  “waiver”  in  the  OTRR  process  continues  to  create  confusion 
among  Department  of  Navy  acquisition  community  leaders  and  staff,  as  well  as 
the  IG  audit  team.  The  desired  action  is  for  the  PEO  to  notify  ASN(RDA)  that  a 
program  is  proceeding  to  OT  without  achieving  one  or  more  of  the  20  specified 
OT  certification  criteria.  It  is  not  a  mechanism  for  N84  to  provide  a  program  with 
relief  from  a  specific  CDD/CPD  Requirement.  The  term  “waiver”  will  be 
replaced  with  “Deviation  from  SECNAV  Policy”.  The  term  “deferral”  will  be 
replaced  with  "Deferral  of  Test  Requirements"  and  applies  to  a  delay  in  testing 
capabilities  identified  in  the  current  requirements  document  and  agreed  to  in  the 
TEMP. 

b.  The  audit  states  that  there  is  no  requirement  to  document  to  the  PEO  how  each  of 
the  criteria  have  been  met.  Additionally,  there  has  been  some  discussion  that  a 
certification  items  could  be  characterized  as  “partially  met”.  To  address  these 
issues,  the  guidance  will  read: 

“Certification  Criteria  assessments  are  binary  -  ‘met’  or  ‘not  met’.  There  is  no 
‘partially  met’. 

OTRR  briefs  shall  include  a  summary  chart  that  characterizes  program  office 
assessment  of  each  of  the  20  criteria  for  lOT&E  (or  the  tailored  set  for  events 
other  than  lOT&E),  and  ensure  that  each  criterion  is  discussed  in  adequate  detail 
to  support  the  program  office  assessment.” 

c.  As  stated  above,  the  SECNAVINST  5000.2E  describes  that  a  PEO  must  request  a 
waiver  when  certification  criteria  are  not  met  in  three  short  paragraphs.  There  is 
not  a  single  sentence  that  states  the  requirement.  To  clarify  this,  the  interim 
guidance  will  state: 

“Deviations  from  SECNAV  Policy  and  Deferral  of  Test  Requirements  shall  be 
discussed  in  the  OTRR  and  formally  requested  in  the  OT&E  Certification 
Message.  If  a  deviation  or  deferral  request  is  anticipated,  the  PM  shall  coordinate 
with  the  program  sponsor  (Navy  only),  CNO  (N84)/DC,  CD&I,  and 
COMOPTEVFOR  /  Director,  MCOTEA  prior  to  the  OTRR.  When  requesting  a 
Deviation  in  the  OTRR  Certification  Message,  the  PEO  shall  identify  which  of 
the  20  certification  criteria  that  are  not  met,  and  state  why  the  decision  was  made 
to  proceed  to  OT&E  without  meeting  those  specific  criteria. 

When  requesting  a  deferral  in  the  OTRR  Certification  Message,  the  PM  shall 
describe  1)  the  capability  limitation  2)  the  operational  impact  upon  the  system 
under  test,  and  3)  the  planned  timeframe  for  testing  the  capability  in  a  future 
OT&E  period.” 
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Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition  Comments  (cont'd) 


NAVY  RESPONSE  TO 

DODIG  DRAFT  AUDIT  REPORT,  “NAVAL  AIR  SYSTEMS  COMMAND  NEEDS  TO 
IMPROVE  MANAGEMENT  OF  WAIVER  REQUESTS”, 
D20I4-D000AE-I64.000,  DATED  12  MARCH  2015 


Finding  B:  Requirements  Documents  Must  Identify  System  Characteristics  Most 
Critical  to  Providing  an  Effective  Military  Capability 

Weapon  system  sponsors  within  the  Office  of  the  Chief  of  Naval  Operations  (CNO)  did 
not  fully  define  the  aircraft  system  characteristics  most  critical  for  providing  an  effective 
military  capability.  In  addition,  they  did  not  designate  the  characteristics  as  key 
performance  parameters  (primary  requirements)  when  writing  system  requirements 
documents  for  the  P-8A  and  E-2D  aircraft  programs.  Additionally,  the  Joint  Chiefs  of 
Staffs  Joint  Requirement  Oversight  Council  (JROC)  review  and  validation  of  those 
requirements  documents  did  not  focus  the  primary  requirements  on  the  accomplishment 
of  critical  aircraft  missions. 

These  conditions  occurred  because  Joint  Chiefs  of  Staff  guidance  did  not  specifically 
require  the  CNO  staff  and  the  JROC  to  make  sure  the  requirements  document  captures 
system  characteristics  that  are  most  critical  to  meeting  mission  requirements  as  primary 
requirements.  As  a  result,  the  weapon  system  sponsors  concurred  with  P-8  A  and  the  E- 
2D  waiver  requests  without  giving  the  Joint  Chiefs  of  Staff  the  opportunity  to  assess  the 
effect  of  those  waivers  on  the  military  usefulness  of  the  aircraft.  Additionally,  because 
the  requirements  documents  did  not  capture  the  most  critical  system  characteristics  as 
primary  requirements,  PMs  accepted  production  units  with  diminished  mission 
effectiveness.  For  example,  at  the  time  of  the  final  production  decision,  the  PMs  for  the 
P-8A  Poseidon  had  accepted  13  aircraft  for  $2.6  billion  that  could  not  fully  perform 
primary  missions  for  Anti-Submarine  Warfare,  Anti-Surface  Warfare,  and  Intelligence 
gathering. 

QPNAV  Response: 

N84/N803:  Do  not  concur 

Finding  B  implies  that  the  only  way  a  PM/PEO  is  informed  of  a  system’s  required  capabilities 
and  determines  if  a  system  is  ready  to  proceed  to  operational  testing  is  through  reading  a  detailed 
requirement  document.  This  incorrectly  assumes  that  the  requirement  document  is  the  only 
vehicle  for  conveying  programmatic  information  to  the  PEO  and  PM.  Resource  sponsors,  PEOs 
and  PMs  are  in  daily  communications  discussing  risk,  cost,  performance  and  schedule. 

Numerous  technical  and  fiscal  realities,  as  well  as  actual  system  performance,  all  affect  the 
decision  making  process  associated  OTRRs. 

Existing  SECNAVINST  Policy  paragraph  4. 6.2.2  states  that  a  PEO  must  execute  the  following 
policy  when  certifying  a  system  to  proceed  to  OT  with  Exceptions  -  which  encompasses  both 
waivers  and  deferrals  (italics  added  for  accent): 
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Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition  Comments  (cont'd) 


NAVY  RESPONSE  TO 

DODIG  DRAFT  AUDIT  REPORT,  “NAVAL  AIR  SYSTEMS  COMMAND  NEEDS  TO 
IMPROVE  MANAGEMENT  OF  WAIVER  REQUESTS”, 
D20I4-D000AE-I64.000,  DATED  12  MARCH  2015 

”4. 6.2.2  Certification  for  OT  With  T&E  Exceptions 

Certify  to  CNO  ((N84),  DC,  CD&I  by  message  that  a  system  is  ready  for  OT _ (specific 

operational  test  phase),  as  required  by  the  TEMP,  with  waiver  and  or  deferral  requests.  Provide 
information  copies  to  the  program  sponsor  (Navy  only,  who  must  provide  formal  concurrence 
with  proposed  exceptions),  ASN(RD&A),  COMOPTEVFOR  and  Director,  MCOTEA,  and  when 
a  program  is  on  the  OSD  T&E  oversight  list,  to  DOT&E.” 

This  policy  ensures  that  the  PEO  delegated  Title  10  responsibility  from  the  CNO  to  develop 
requirements  for  systems  that  will  equip  the  fleet  have  executed  that  responsibility  in 
determining  that  the  system  capabilities  are  adequate  to  proceed  to  operational  testing.  The  Navy 
views  this  as  adequate  oversight. 

Adding  a  requirement  for  a  waiver  to  be  approved  by  the  JROC,  would  also  force  many  Flag  level 
briefs  -  up  to  the  four  star  level  -  before  the  waiver  would  even  leave  the  Navy  to  start  Joint  Staffing. 
This  is  a  significant  administrative  burden  that  is  contradictory  to  CNO  direction  and  USD  AT&L 
BBP  3.0  guidance  and  would  ultimately  impact  program  execution. 

It  is  also  noted  that  the  DODIG  audit  team  chose  to  ignore  OPNAV  N84’s  recommendations  to 
interview  the  owners  of  the  Navy  Requirements  process  -  CNO  JCIDS  gatekeeper  staff  (OPNAV 
N803)  and  the  Requirements  Officers  (OPNAV  N98  for  P-8/DTS  and  OPNAV  N2/6  for  E-2D), 
so  they  would  better  understand  and  characterize  their  role  in  this  process. 

NAY  AIR  Response: 

PMA-290:  Do  not  concur 

Do  not  concur  with  the  finding  that  Weapon  System  sponsors  within  the  Office  of  the  CNO  did 
not  designate  characteristics  most  critical  to  providing  an  effective  military  capability  as  primary 
requirements  when  writing  aircraft  requirements  documents.  The  Fleet's  most  urgent  need  for 
the  P-8A  platform  was  for  the  Anti-Submarine  Warfare  (ASW)  mission  conducted  by  the  aging 
P-3C  Orion.  Mission  systems  for  ASW  and  other  missions  were  derivatives  of  mature 
technologies  that  were  successfully  performing  the  same  missions  on  the  P-3C.  As  such,  the 
requirements  were  logically  written  to  define  the  needed  replacement  platform,  and  platform 
performance  was  appropriately  designated  as  one  of  the  most  critical  characteristics  of  the  P-8A 
as  a  platform  replacement  program. 

Recommendation  B.l:  We  recommend  that  Vice  Chairman,  Joint  Chiefs  of  Staff,  modify  the 
Joint  Capabilities  Integration  and  Development  System  (JCIDS)  Manual  to  add  clarifying 
language  regarding  the  need  for: 

a.  sponsors  to  define  those  system  characteristics  that  are  most  critical  for 
providing  an  effective  military  capability  and  designate  them  as  primary 
requirements  when  writing  requirements  documents;  and 
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NAVY  RESPONSE  TO 

DODIG  DRAFT  AUDIT  REPORT,  “NAVAL  AIR  SYSTEMS  COMMAND  NEEDS  TO 
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D20I4-D000AE-I64.000,  DATED  12  MARCH  2015 


b.  Joint  Requirements  Oversight  Council  review  to  make  sure  the  system 

characteristics  most  critical  to  mission  effectiveness  are  captured  as  primary 
requirements. 

QPNAV  Response: 

N84/N803:  Concur 

This  is  already  incorporated  in  the  existing  JCIDS  development,  staffing  and  review  process.  No 
impact  to  existing  process  or  Navy  policy. 


Recommendation  B.2:  We  recommend  that  the  Secretary  of  the  Navy  update  SECNAV 
Instruction  5000.2.E,  “Department  of  the  Navy  Implementation  and  Operation  of  the  Defense 
Acquisition  System  and  the  Joint  Capabilities  Integration  and  Development  System,”  September 
1,  201 1,  to  require  sponsors  of  acquisition  programs  designated  as  Acquisition  Category  I,  or  of 
interest  to  the  Joint  Capability  Board  or  the  Joint  Requirements  Oversight  Council,  to  certify  to 
the  Vice  Chairman,  Joint  Chiefs  of  Staff  that: 

a.  approved  waivers  do  not  adversely  impact  primary  system  requirements;  or; 

b.  if  a  proposed  waiver  does  adversely  impact  primary  system  requirements, 
that  the  system  continues  to  have  military  utility  as  described  in  the  “Manual 
for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development 
System,”  January  10,  2012  before  approving  the  waiver. 


QPNAV  Response: 

N803/N84:  Do  not  concur 

Services  have  the  Title  10  authority  to  man,  train,  and  equip.  JROC  has  Title  10  authority  over 
requirements,  not  acquisition  implementation  of  them.  Once  the  JROC  validates  the 
requirements,  it  becomes  the  CNO’s  responsibility  to  ensure  they  are  met,  and  ASN(RDA)’s 
responsibility  to  acquire  the  appropriate  material  solution.  The  PEO,  as  ASN(RDA)’s  executive 
agent,  in  consultation  with  the  CNO  staff,  is  responsible  for  assessing  whether  a  system  shortfall 
impacts  meeting  CDD/CPD  requirements  when  certifying  a  system  to  proceed  to  OT.  Existing 
SECNAVINST  Policy  paragraph  4. 6.2.2  states  that  a  PEO  must  follow  the  following  policy 
when  certifying  a  system  to  proceed  to  OT  with  Exceptions  -  which  encompasses  both  waivers 
and  deferrals  (italics  added  for  accent): 

’’Certification  for  OT  With  T&E  Exceptions 
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DODIG  DRAFT  AUDIT  REPORT,  “NAVAL  AIR  SYSTEMS  COMMAND  NEEDS  TO 
IMPROVE  MANAGEMENT  OF  WAIVER  REQUESTS”, 
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Certify  to  CNO  (N84),  DC,  CD&I  by  message  that  a  system  is  ready  for  OT _ (specific 

operational  test  phase),  as  required  by  the  TEMP,  with  waiver  and  or  deferral  requests.  Provide 
information  copies  to  the  program  sponsor  (Navy  only,  who  must  provide  formal  concurrence 
with  proposed  exceptions),  ASN(RD&A),  COMOPTEVFOR  and  Director,  MCOTEA,  and  when 
a  program  is  on  the  OSD  T&E  oversight  list,  to  DOT&E.” 

This  policy  ensures  that  the  flag  officer  delegated  Title  10  responsibility  from  the  CNO  to 
develop  requirements  for  systems  that  will  equip  the  fleet  have  executed  that  responsibility  in 
determining  that  the  system  capabilities  are  adequate  to  proceed  to  Operational  Testing. 

Adding  a  requirement  for  a  waiver  to  be  approved  by  the  JROC,  would  also  drive  many  Flag 
level  briefs  -  up  to  the  four  star  level  -  before  the  waiver  would  even  leave  the  Navy  to  start  Joint 
Staffing. 


NAY  AIR  Response: 

PMA-290:  Do  not  concur 

The  current  waiver  and  deferral  process  has  historically  worked  well  for  the  P-8A  program  and 
for  all  PEO(A)  programs.  The  current  structure  and  decision  level  for  waivers  is  effective  for 
ensuring  a  comprehensive  and  efficient  process  for  managing  Acquisition  Category  I  programs. 
The  DoD  IG  proposal  would  effectively  require  Major  Defense  Acquisition  Programs  to 
regularly  elevate  discussions  of  narrowly  scoped  individual  deficiencies  with  minimal  impact  on 
top-level  system  performance  to  the  JROC  or  JCS.  This  would  at  best  create  a  significant,  non¬ 
value  added  administrative  burden. 


General  Comments: 

PMA-290: 

As  discussed  in  this  response,  the  decision  to  enter  lOT&E  was  the  result  of  a  robust 
review  and  risk  assessment  by  key  stakeholders  of  the  deficiencies  at  the  time.  The 
deficiencies  were  assessed  to  be  low  risk  and  not  to  the  degree  to  delay  the  start  of  test 
and  subsequently  the  fielding  of  this  critical  capability  to  the  fleet.  The  decision  was 
validated  by  the  final  lOT&E  determination  that  the  platform  was  Operationally 
Effective,  Operationally  Suitable,  and  Recommended  for  Fleet  Introduction,  and  again  by 
the  P-8A's  considerable  operational  successes  during  its  first  two  operational 
deployments.  It  is  especially  worth  noting  that  frequent  Fleet  feedback  confirms  the 
long-standing  conclusion  of  the  entire  Navy  test  community  that  the  aircraft's  ASW 
systems  are  a  substantial  leap  forward  over  all  previously  fielded  P-3C  systems  even  for 
P-8A  Increment  I ,  and  continue  to  add  capability  in  accordance  with  a  very  carefully 
planned  evolutionary  upgrade  strategy  that  intelligently  balances  cost,  schedule  and 
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DODIG  DRAFT  AUDIT  REPORT,  “NAVAL  AIR  SYSTEMS  COMMAND  NEEDS  TO 
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D20I4-D000AE-I64.000,  DATED  12  MARCH  2015 

performance  without  assuming  unnecessary  technical  risk.  Furthermore,  the  program's 
incremental  requirements  definition  strategy  is  one  that  should  be  replicated  for  future 
programs  if  DoD  and  the  Navy  are  to  efficiently  and  cost-effectively  deliver  new 
capabilities  in  a  resource  constrained  environment. 


Classification  Review: 

Navy:  No  classification  issues  noted. 
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Acronyms  and  Abbreviations 


AHE 

Advanced  Hawkeye 

ASN(RDA) 

Assistant  Secretary  of  the  Navy  for  Research,  Developnnent,  and  Acquisition 

CNO 

Chief  of  Naval  Operations 

DTS 

Distributed  Targeting  Systenn 

lOT&E 

Initial  Operational  Test  and  Evaluation 

JCIDS 

Joint  Capabilities  Integration  and  Developnnent  Systenn 

JCS 

Joint  Chiefs  of  Staff 

JITC 

Joint  Interoperability  Test  Connnnand 

JROC 

Joint  Requirements  Oversight  Council 

NAVAIR 

Naval  Air  Systems  Command 

OTRR 

Operational  Test  Readiness  Review 

PEO 

Program  Executive  Officer 

SECNAVINST 

Secretary  of  the  Navy  Instruction 

TEMP 

Test  and  Evaluation  Master  Plan 
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Whistleblower  Protection 

U.S.  Department  of  Defense 

The  Whistleblower  Protection  Enhancement  Act  of  2012  requires 
the  Inspector  General  to  designate  a  Whistleblower  Protection 
Ombudsman  to  educate  agency  employees  about  prohibitions 
on  retaliation,  and  rights  and  remedies  against  retaliation  for 
protected  disclosures.  The  designated  ombudsman  is  the  DoD  Hotline 
Director.  For  more  information  on  your  rights  and  remedies  against 
retaliation,  visit  www.dodig.mil/programs/whistleblower. 


For  more  information  about  DoD  IG 
reports  or  activities,  please  contact  us: 

Congressional  Liaison 

congressional@dodig.mil;  703.604.8324 

Media  Contact 

public.affairs@dodig.mil;  703.604.8324 

Monthly  Update 

dodigconnect-request@listserve.com 

Reports  Mailing  List 

dodig_report@listserve.com 

Twitter 

twitter.com/DoD_IG 

DoD  Hotline 

dodig.mil/hotline 
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